[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:34:29 dist-test-slave-drl0 krb5kdc[16282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:34:29 dist-test-slave-drl0 krb5kdc[16282](info): set up 2 sockets
Aug 08 04:34:29 dist-test-slave-drl0 krb5kdc[16282](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 04:34:30.333760 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.337s user 0.005s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:34:30 dist-test-slave-drl0 krb5kdc[16282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:34:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:34:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:34:30.492537 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:33331
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:41063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:33331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:30.654892 16299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:30.655648 16299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:30.655714 16299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:30.661407 16299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:34:30.661487 16299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:30.661510 16299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:30.661530 16299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:34:30.661545 16299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:34:30.669121 16299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:41063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:33331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:33331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:30.670729 16299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:30.672161 16299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:30.687775 16306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:30.687898 16311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:30.688069 16299 server_base.cc:1034] running on GCE node
W20260808 04:34:30.688948 16307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:30.689903 16299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:30.691973 16299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:30.695363 16299 hybrid_clock.cc:648] HybridClock initialized: now 1786163670695304 us; error 121 us; skew 500 ppm
Aug 08 04:34:30 dist-test-slave-drl0 krb5kdc[16282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163670, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:30.698859 16299 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:34:30.700223 16299 webserver.cc:460] Webserver started at http://127.15.228.126:33933/ using document root <none> and password file <none>
I20260808 04:34:30.701220 16299 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:30.701327 16299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:30.701767 16299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:30.704931 16299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "fe87909f67514830ba2e74f74127a911"
format_stamp: "Formatted at 2026-08-08 04:34:30 on dist-test-slave-drl0"
server_key: "cd245ebe7063bcb0b1375cb517676bd6"
server_key_iv: "976126e685d5621e4928a6b2051f1e84"
server_key_version: "encryptionkey@0"
I20260808 04:34:30.705804 16299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "fe87909f67514830ba2e74f74127a911"
format_stamp: "Formatted at 2026-08-08 04:34:30 on dist-test-slave-drl0"
server_key: "cd245ebe7063bcb0b1375cb517676bd6"
server_key_iv: "976126e685d5621e4928a6b2051f1e84"
server_key_version: "encryptionkey@0"
I20260808 04:34:30.712359 16299 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 04:34:30.716915 16324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:30.718819 16299 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:34:30.719082 16299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "fe87909f67514830ba2e74f74127a911"
format_stamp: "Formatted at 2026-08-08 04:34:30 on dist-test-slave-drl0"
server_key: "cd245ebe7063bcb0b1375cb517676bd6"
server_key_iv: "976126e685d5621e4928a6b2051f1e84"
server_key_version: "encryptionkey@0"
I20260808 04:34:30.719229 16299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:30.737408 16299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:30.741597 16299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:30.741917 16299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:30.754812 16299 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:33331
I20260808 04:34:30.754864 16404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:33331 every 8 connection(s)
I20260808 04:34:30.756958 16299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:34:30.762923 16405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:30.766244 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 16299
I20260808 04:34:30.766435 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:34:30.769586 16273 external_mini_cluster.cc:1428] Setting key e70e74945a49969a9b1d769f3d4d41fc
I20260808 04:34:30.777719 16405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911: Bootstrap starting.
I20260808 04:34:30.782003 16405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:30.783334 16405 log.cc:824] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:30.787009 16405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911: No bootstrap required, opened a new log
I20260808 04:34:30.794602 16405 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe87909f67514830ba2e74f74127a911" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33331 } }
I20260808 04:34:30.795269 16405 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:30.795445 16405 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe87909f67514830ba2e74f74127a911, State: Initialized, Role: FOLLOWER
I20260808 04:34:30.796324 16405 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe87909f67514830ba2e74f74127a911" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33331 } }
I20260808 04:34:30.796558 16405 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:34:30.796706 16405 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:34:30.796868 16405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:30.798137 16405 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe87909f67514830ba2e74f74127a911" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33331 } }
Aug 08 04:34:30 dist-test-slave-drl0 krb5kdc[16282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:30.798892 16405 leader_election.cc:302] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe87909f67514830ba2e74f74127a911; no voters:
I20260808 04:34:30.800021 16405 leader_election.cc:288] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:34:30.800187 16411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:34:30.800797 16411 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911 [term 1 LEADER]: Becoming Leader. State: Replica: fe87909f67514830ba2e74f74127a911, State: Running, Role: LEADER
I20260808 04:34:30.801473 16411 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe87909f67514830ba2e74f74127a911" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33331 } }
I20260808 04:34:30.802214 16405 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:34:30.805243 16414 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe87909f67514830ba2e74f74127a911. Latest consensus state: current_term: 1 leader_uuid: "fe87909f67514830ba2e74f74127a911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe87909f67514830ba2e74f74127a911" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33331 } } }
I20260808 04:34:30.805424 16414 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911 [sys.catalog]: This master's current role is: LEADER
I20260808 04:34:30.805272 16413 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe87909f67514830ba2e74f74127a911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe87909f67514830ba2e74f74127a911" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33331 } } }
I20260808 04:34:30.805824 16413 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911 [sys.catalog]: This master's current role is: LEADER
I20260808 04:34:30.808966 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:30.777341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.15.228.126:33331)
0808 04:34:30.778066 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:30.778086 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:30.781806 (+ 3720us) server_negotiation.cc:407] Connection header received
0808 04:34:30.785718 (+ 3912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:30.785748 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:30.786650 (+ 902us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:30.787269 (+ 619us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:30.790289 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:30.792024 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:30.793719 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:30.795072 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:30.795319 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:30.799862 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:30.799908 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:30.799921 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:30.799956 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:30.802594 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:30.803295 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:30.803304 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:30.803310 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:30.803545 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:30.804070 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:30.804074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:30.804075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:30.804767 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:30.805033 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:30.807523 (+ 2490us) server_negotiation.cc:299] Negotiation successful
0808 04:34:30.807893 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":210,"threads_started":1}
W20260808 04:34:30.812350 16431 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:34:30.812850 16431 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:34:30.813079 16433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:34:30.814671 16433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:34:30.824688 16433 catalog_manager.cc:1347] Generated new cluster ID: 0801b9d0a1bb4b27aed606a7e2fe143e
I20260808 04:34:30.824788 16433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:34:30.845468 16433 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:34:30.847040 16433 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:34:30.863775 16433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe87909f67514830ba2e74f74127a911: Generated new TSK 0
I20260808 04:34:30.865269 16433 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
I20260808 04:34:30.914857 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33331
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:41063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:31.080209 16441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:31.080677 16441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:31.080775 16441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:31.086022 16441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:31.086203 16441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:31.086344 16441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:34:31.092893 16441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:41063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33331
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:31.094159 16441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:31.095359 16441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:31.107225 16447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:31.107231 16448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:31.107225 16450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:31.107822 16441 server_base.cc:1034] running on GCE node
I20260808 04:34:31.108505 16441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:31.109581 16441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:31.110911 16441 hybrid_clock.cc:648] HybridClock initialized: now 1786163671110925 us; error 68 us; skew 500 ppm
Aug 08 04:34:31 dist-test-slave-drl0 krb5kdc[16282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163671, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:31.115284 16441 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:34:31.116420 16441 webserver.cc:460] Webserver started at http://127.15.228.65:43909/ using document root <none> and password file <none>
I20260808 04:34:31.117058 16441 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:31.117153 16441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:31.117405 16441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:31.119777 16441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "f3553fc7733048e4bf3a11903d69f188"
format_stamp: "Formatted at 2026-08-08 04:34:31 on dist-test-slave-drl0"
server_key: "c4a6dad3b6da95484f85d16362243ae0"
server_key_iv: "59e6e353b4108b3f6bd28a97704860b4"
server_key_version: "encryptionkey@0"
I20260808 04:34:31.120404 16441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "f3553fc7733048e4bf3a11903d69f188"
format_stamp: "Formatted at 2026-08-08 04:34:31 on dist-test-slave-drl0"
server_key: "c4a6dad3b6da95484f85d16362243ae0"
server_key_iv: "59e6e353b4108b3f6bd28a97704860b4"
server_key_version: "encryptionkey@0"
I20260808 04:34:31.124939 16441 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:34:31.127890 16461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:31.129685 16441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:34:31.129863 16441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "f3553fc7733048e4bf3a11903d69f188"
format_stamp: "Formatted at 2026-08-08 04:34:31 on dist-test-slave-drl0"
server_key: "c4a6dad3b6da95484f85d16362243ae0"
server_key_iv: "59e6e353b4108b3f6bd28a97704860b4"
server_key_version: "encryptionkey@0"
I20260808 04:34:31.130007 16441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:31.152289 16441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:31.156895 16441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:31.157161 16441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:31.157967 16441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:34:31.159152 16441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:31.159229 16441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:31.159298 16441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:31.159343 16441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:31.177706 16441 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:36233
I20260808 04:34:31.177762 16626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:36233 every 8 connection(s)
I20260808 04:34:31.178715 16441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:34:31.179100 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 16441
I20260808 04:34:31.179193 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:34:31.179469 16273 external_mini_cluster.cc:1428] Setting key ee8cf0f99cf0bf6265affb49480e10ca
WARNING: no policy specified for kudu/127.15.228.66@KRBTEST.COM; defaulting to no policy
Aug 08 04:34:31 dist-test-slave-drl0 krb5kdc[16282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163671, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.66@KRBTEST.COM" created.
I20260808 04:34:31.199709 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:31.181397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:38791 (local address 127.15.228.126:33331)
0808 04:34:31.181533 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:31.181537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:31.183004 (+ 1467us) server_negotiation.cc:407] Connection header received
0808 04:34:31.184371 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:31.184375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:31.184430 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:31.184543 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:31.185910 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:31.186596 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:31.188992 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:31.189285 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:31.189428 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:31.194382 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:31.194387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:31.194390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:31.194435 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:31.196437 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:31.197451 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:31.197456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:31.197458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:31.197589 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:31.198112 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:31.198124 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:31.198128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:31.198449 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:31.198610 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:31.199287 (+ 677us) server_negotiation.cc:299] Negotiation successful
0808 04:34:31.199514 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:34:31.201069 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:31.181983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:33331 (local address 127.15.228.65:38791)
0808 04:34:31.182811 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:31.182860 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:31.184134 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:31.184725 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:31.184731 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:31.185117 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:31.185748 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:31.185763 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:31.186920 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:31.187078 (+ 158us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:31.188717 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:31.188745 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:31.189520 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:31.189525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:31.189747 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:31.190733 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:31.190756 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:31.194002 (+ 3246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:31.196667 (+ 2665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:31.196679 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:31.196711 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:31.197258 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:31.197746 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:31.197750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:31.197752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:31.197960 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:31.198609 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:31.198616 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:31.198912 (+ 296us) client_negotiation.cc:769] Sending connection context
0808 04:34:31.199187 (+ 275us) client_negotiation.cc:240] Negotiation successful
0808 04:34:31.199676 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":145,"threads_started":1}
I20260808 04:34:31.202674 16627 heartbeater.cc:347] Connected to a master server at 127.15.228.126:33331
I20260808 04:34:31.203007 16627 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:31.203650 16627 heartbeater.cc:511] Master 127.15.228.126:33331 requested a full tablet report, sending...
I20260808 04:34:31.206176 16347 ts_manager.cc:194] Registered new tserver with Master: f3553fc7733048e4bf3a11903d69f188 (127.15.228.65:36233)
I20260808 04:34:31.208593 16347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:38791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
I20260808 04:34:31.219770 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33331
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:41063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:31.380467 16633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:31.380748 16633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:31.380795 16633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:31.385183 16633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:31.385246 16633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:31.385308 16633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:34:31.390828 16633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:41063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33331
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:31.392174 16633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:31.394292 16633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:31.408039 16640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:31.408082 16639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:31.408205 16642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:31.408715 16633 server_base.cc:1034] running on GCE node
I20260808 04:34:31.409307 16633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:31.410269 16633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:31.411582 16633 hybrid_clock.cc:648] HybridClock initialized: now 1786163671411556 us; error 35 us; skew 500 ppm
Aug 08 04:34:31 dist-test-slave-drl0 krb5kdc[16282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163671, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:31.415676 16633 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:34:31.416937 16633 webserver.cc:460] Webserver started at http://127.15.228.66:37509/ using document root <none> and password file <none>
I20260808 04:34:31.417621 16633 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:31.417691 16633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:31.417973 16633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:31.420532 16633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "aac9118ca0134cd5b6485a37ba04a10e"
format_stamp: "Formatted at 2026-08-08 04:34:31 on dist-test-slave-drl0"
server_key: "b55f27ab62ad1a4cd1875b8ff0dde7af"
server_key_iv: "b39a302fde8a38a3704e2d3d797da47f"
server_key_version: "encryptionkey@0"
I20260808 04:34:31.421233 16633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "aac9118ca0134cd5b6485a37ba04a10e"
format_stamp: "Formatted at 2026-08-08 04:34:31 on dist-test-slave-drl0"
server_key: "b55f27ab62ad1a4cd1875b8ff0dde7af"
server_key_iv: "b39a302fde8a38a3704e2d3d797da47f"
server_key_version: "encryptionkey@0"
I20260808 04:34:31.426616 16633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:34:31.430320 16650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:31.432080 16633 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:34:31.432261 16633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "aac9118ca0134cd5b6485a37ba04a10e"
format_stamp: "Formatted at 2026-08-08 04:34:31 on dist-test-slave-drl0"
server_key: "b55f27ab62ad1a4cd1875b8ff0dde7af"
server_key_iv: "b39a302fde8a38a3704e2d3d797da47f"
server_key_version: "encryptionkey@0"
I20260808 04:34:31.432408 16633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:31.449230 16633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:31.453639 16633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:31.453925 16633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:31.454732 16633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:34:31.455765 16633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:31.455814 16633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:31.455858 16633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:31.455926 16633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:31.472112 16633 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:32883
I20260808 04:34:31.472474 16813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:32883 every 8 connection(s)
I20260808 04:34:31.473337 16633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:34:31.481266 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 16633
I20260808 04:34:31.481411 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:34:31.481686 16273 external_mini_cluster.cc:1428] Setting key 9f750d8148873066fbad71a5daf7cd85
Aug 08 04:34:31 dist-test-slave-drl0 krb5kdc[16282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163671, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:31.491274 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:31.476242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:55651 (local address 127.15.228.126:33331)
0808 04:34:31.476445 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:31.476449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:31.477714 (+ 1265us) server_negotiation.cc:407] Connection header received
0808 04:34:31.478680 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:31.478683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:31.478759 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:31.478843 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:31.480329 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:31.480896 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:31.482519 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:31.482767 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:31.482895 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:31.486639 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:31.486644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:31.486647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:31.486693 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:31.488992 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:31.489594 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:31.489599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:31.489601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:31.489692 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:31.490064 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:31.490069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:31.490071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:31.490267 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:31.490384 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:31.490916 (+ 532us) server_negotiation.cc:299] Negotiation successful
0808 04:34:31.491101 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260808 04:34:31.492143 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:31.476618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:33331 (local address 127.15.228.66:55651)
0808 04:34:31.477488 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:31.477533 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:31.478429 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:31.479114 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:31.479122 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:31.479584 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:31.480158 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:31.480173 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:31.481106 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:31.481109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:31.482384 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:31.482395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:31.482993 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:31.483000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:31.483163 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:31.483811 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:31.483829 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:31.486176 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:31.489150 (+ 2974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:31.489157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:31.489174 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:31.489474 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:31.489796 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:31.489802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:31.489806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:31.489959 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:31.490384 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:31.490393 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:31.490658 (+ 265us) client_negotiation.cc:769] Sending connection context
0808 04:34:31.490860 (+ 202us) client_negotiation.cc:240] Negotiation successful
0808 04:34:31.491270 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":220,"threads_started":1}
I20260808 04:34:31.493489 16814 heartbeater.cc:347] Connected to a master server at 127.15.228.126:33331
I20260808 04:34:31.494055 16814 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:31.495119 16814 heartbeater.cc:511] Master 127.15.228.126:33331 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.228.67@KRBTEST.COM; defaulting to no policy
I20260808 04:34:31.496922 16347 ts_manager.cc:194] Registered new tserver with Master: aac9118ca0134cd5b6485a37ba04a10e (127.15.228.66:32883)
I20260808 04:34:31.497753 16347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:55651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
I20260808 04:34:31.518875 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33331
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:41063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:31.668265 16822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:31.668576 16822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:31.668635 16822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:31.673152 16822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:31.673207 16822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:31.673266 16822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:34:31.679123 16822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:41063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33331
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:31.680392 16822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:31.681746 16822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:31.695389 16830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:31.695405 16828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:31.695447 16827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:31.695487 16822 server_base.cc:1034] running on GCE node
I20260808 04:34:31.696298 16822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:31.697081 16822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:31.698305 16822 hybrid_clock.cc:648] HybridClock initialized: now 1786163671698268 us; error 49 us; skew 500 ppm
Aug 08 04:34:31 dist-test-slave-drl0 krb5kdc[16282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163671, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:31.702091 16822 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:34:31.703099 16822 webserver.cc:460] Webserver started at http://127.15.228.67:38529/ using document root <none> and password file <none>
I20260808 04:34:31.703747 16822 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:31.703840 16822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:31.704088 16822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:31.706585 16822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "ce66c9aafe2d439aafa846c66527aa6b"
format_stamp: "Formatted at 2026-08-08 04:34:31 on dist-test-slave-drl0"
server_key: "619b182f6ccbbb73b57ba744c0473288"
server_key_iv: "08fd256c2f28e9615229fe93395ab768"
server_key_version: "encryptionkey@0"
I20260808 04:34:31.707309 16822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "ce66c9aafe2d439aafa846c66527aa6b"
format_stamp: "Formatted at 2026-08-08 04:34:31 on dist-test-slave-drl0"
server_key: "619b182f6ccbbb73b57ba744c0473288"
server_key_iv: "08fd256c2f28e9615229fe93395ab768"
server_key_version: "encryptionkey@0"
I20260808 04:34:31.713860 16822 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260808 04:34:31.717752 16839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:31.719400 16822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:34:31.719563 16822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "ce66c9aafe2d439aafa846c66527aa6b"
format_stamp: "Formatted at 2026-08-08 04:34:31 on dist-test-slave-drl0"
server_key: "619b182f6ccbbb73b57ba744c0473288"
server_key_iv: "08fd256c2f28e9615229fe93395ab768"
server_key_version: "encryptionkey@0"
I20260808 04:34:31.719705 16822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:31.745088 16822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:31.749269 16822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:31.749531 16822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:31.750309 16822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:34:31.751595 16822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:31.751699 16822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:31.751781 16822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:31.751827 16822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:31.771409 16822 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:34601
I20260808 04:34:31.771940 16998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:34601 every 8 connection(s)
I20260808 04:34:31.772897 16822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:34:31.780737 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 16822
I20260808 04:34:31.780872 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:34:31.781205 16273 external_mini_cluster.cc:1428] Setting key 4bb1320546e191599f518d6eea6d18a2
Aug 08 04:34:31 dist-test-slave-drl0 krb5kdc[16282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163671, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:31.790907 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:31.775340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:39163 (local address 127.15.228.126:33331)
0808 04:34:31.775505 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:31.775511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:31.776480 (+ 969us) server_negotiation.cc:407] Connection header received
0808 04:34:31.777660 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:31.777664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:31.777750 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:31.777877 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:31.779200 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:31.780158 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:31.781988 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:31.782477 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:31.782630 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:31.786439 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:31.786443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:31.786445 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:31.786487 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:31.788350 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:31.789063 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:31.789067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:31.789068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:31.789232 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:31.789667 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:31.789671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:31.789672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:31.789949 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:31.790037 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:31.790639 (+ 602us) server_negotiation.cc:299] Negotiation successful
0808 04:34:31.790765 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:34:31.791764 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:31.775750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:33331 (local address 127.15.228.67:39163)
0808 04:34:31.776309 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:31.776357 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:31.777338 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:31.778069 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:31.778075 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:31.778500 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:31.779035 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:31.779049 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:31.780421 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:31.780424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:31.781708 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:31.781724 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:31.782841 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:31.782847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:31.783104 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:31.783811 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:31.783828 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:31.786209 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:31.788548 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:31.788554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:31.788574 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:31.788898 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:31.789391 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:31.789394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:31.789395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:31.789541 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:31.790077 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:31.790082 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:31.790357 (+ 275us) client_negotiation.cc:769] Sending connection context
0808 04:34:31.790547 (+ 190us) client_negotiation.cc:240] Negotiation successful
0808 04:34:31.790821 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":155,"threads_started":1}
I20260808 04:34:31.793222 16999 heartbeater.cc:347] Connected to a master server at 127.15.228.126:33331
I20260808 04:34:31.793648 16999 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:31.794371 16999 heartbeater.cc:511] Master 127.15.228.126:33331 requested a full tablet report, sending...
I20260808 04:34:31.795890 16347 ts_manager.cc:194] Registered new tserver with Master: ce66c9aafe2d439aafa846c66527aa6b (127.15.228.67:34601)
I20260808 04:34:31.796480 16347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:39163
I20260808 04:34:31.799103 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 04:34:31 dist-test-slave-drl0 krb5kdc[16282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163671, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 04:34:31 dist-test-slave-drl0 krb5kdc[16282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163671, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.228.65@KRBTEST.COM
I20260808 04:34:31.836792 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:31.822574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53004 (local address 127.15.228.65:36233)
0808 04:34:31.823028 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:31.823039 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:31.823085 (+ 46us) server_negotiation.cc:407] Connection header received
0808 04:34:31.823284 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:31.823308 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:31.823624 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:31.823818 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:31.824696 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:31.825713 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:31.827068 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:31.827589 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:31.827744 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:31.831478 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:31.831556 (+ 78us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:31.831575 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:31.831617 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:31.833891 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:31.834628 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:31.834635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:31.834637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:31.834751 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:31.835377 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:31.835381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:31.835383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:31.835781 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:31.836077 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:31.836341 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 04:34:31.836538 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":95,"threads_started":1}
W20260808 04:34:31.838389 16496 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53004
Aug 08 04:34:31 dist-test-slave-drl0 krb5kdc[16282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 04:34:31 dist-test-slave-drl0 krb5kdc[16282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.65@KRBTEST.COM
I20260808 04:34:31.871256 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:31.861227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53008 (local address 127.15.228.65:36233)
0808 04:34:31.861414 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:31.861419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:31.861436 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:34:31.861564 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:31.861566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:31.861616 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:31.861691 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:31.862447 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:31.863161 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:31.864047 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:31.864366 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:31.864474 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:31.867400 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:31.867403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:31.867405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:31.867429 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:31.869002 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:31.869704 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:31.869706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:31.869708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:31.869784 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:31.870203 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:31.870206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:31.870207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:31.870388 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:31.870512 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:31.870847 (+ 335us) server_negotiation.cc:299] Negotiation successful
0808 04:34:31.871045 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 04:34:31.872970 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 16441
I20260808 04:34:31.884907 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 16633
I20260808 04:34:31.904647 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 16822
I20260808 04:34:31.919699 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 16299
2026-08-08T04:34:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2967 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:34:31 dist-test-slave-drl0 krb5kdc[17016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:34:31 dist-test-slave-drl0 krb5kdc[17016](info): set up 2 sockets
Aug 08 04:34:31 dist-test-slave-drl0 krb5kdc[17016](info): commencing operation
krb5kdc: starting...
W20260808 04:34:33.193938 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.243s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:34:33 dist-test-slave-drl0 krb5kdc[17016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:34:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:34:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:34:33.339649 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:45999
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:38613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:45999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:33.465211 17034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:33.465499 17034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:33.465541 17034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:33.469910 17034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:34:33.469975 17034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:33.469995 17034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:33.470012 17034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:34:33.470028 17034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:34:33.476418 17034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:38613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:45999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:45999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:33.478202 17034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:33.479697 17034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:33.491444 17049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:33.491341 17044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:33.491712 17034 server_base.cc:1034] running on GCE node
W20260808 04:34:33.491341 17045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:33.492779 17034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:33.494701 17034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:33.495976 17034 hybrid_clock.cc:648] HybridClock initialized: now 1786163673495934 us; error 64 us; skew 500 ppm
Aug 08 04:34:33 dist-test-slave-drl0 krb5kdc[17016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163673, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:33.499766 17034 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:34:33.500859 17034 webserver.cc:460] Webserver started at http://127.15.228.126:39673/ using document root <none> and password file <none>
I20260808 04:34:33.501519 17034 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:33.501619 17034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:33.501866 17034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:33.504149 17034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "5407985b422a478f94ecec6cb7faac97"
format_stamp: "Formatted at 2026-08-08 04:34:33 on dist-test-slave-drl0"
server_key: "d9f6dffddfdd63d4ac63fc2b59165217"
server_key_iv: "83e26792b060b17a906e18489cd9b9d6"
server_key_version: "encryptionkey@0"
I20260808 04:34:33.504761 17034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "5407985b422a478f94ecec6cb7faac97"
format_stamp: "Formatted at 2026-08-08 04:34:33 on dist-test-slave-drl0"
server_key: "d9f6dffddfdd63d4ac63fc2b59165217"
server_key_iv: "83e26792b060b17a906e18489cd9b9d6"
server_key_version: "encryptionkey@0"
I20260808 04:34:33.508929 17034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 04:34:33.512187 17062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:33.514343 17034 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:34:33.514531 17034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "5407985b422a478f94ecec6cb7faac97"
format_stamp: "Formatted at 2026-08-08 04:34:33 on dist-test-slave-drl0"
server_key: "d9f6dffddfdd63d4ac63fc2b59165217"
server_key_iv: "83e26792b060b17a906e18489cd9b9d6"
server_key_version: "encryptionkey@0"
I20260808 04:34:33.514665 17034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:33.530984 17034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:33.535430 17034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:33.535737 17034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:33.545920 17034 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:45999
I20260808 04:34:33.545977 17141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:45999 every 8 connection(s)
I20260808 04:34:33.547358 17034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:34:33.549014 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 17034
I20260808 04:34:33.549171 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:34:33.549484 16273 external_mini_cluster.cc:1428] Setting key f3dcf5d7f5f749fe8649d601733c783d
I20260808 04:34:33.551529 17142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:33.559844 17142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97: Bootstrap starting.
Aug 08 04:34:33 dist-test-slave-drl0 krb5kdc[17016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:33.563611 17142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:33.564616 17142 log.cc:824] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:33.567392 17142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97: No bootstrap required, opened a new log
I20260808 04:34:33.568037 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:33.551151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44082 (local address 127.15.228.126:45999)
0808 04:34:33.551760 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:33.551777 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:33.551815 (+ 38us) server_negotiation.cc:407] Connection header received
0808 04:34:33.552491 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:33.552507 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:33.552856 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:33.553218 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:33.554410 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:33.555795 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:33.556831 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:33.557433 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:33.557638 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:33.561035 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:33.561042 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:33.561061 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:33.561104 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:33.563993 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:33.564559 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:33.564565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:33.564571 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:33.564731 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:33.565059 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:33.565063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:33.565064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:33.566004 (+ 940us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:33.566391 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:33.566773 (+ 382us) server_negotiation.cc:299] Negotiation successful
0808 04:34:33.567140 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":187,"threads_started":1}
I20260808 04:34:33.572326 17142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5407985b422a478f94ecec6cb7faac97" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45999 } }
I20260808 04:34:33.572566 17142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:33.572602 17142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5407985b422a478f94ecec6cb7faac97, State: Initialized, Role: FOLLOWER
I20260808 04:34:33.573067 17142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5407985b422a478f94ecec6cb7faac97" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45999 } }
I20260808 04:34:33.573194 17142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:34:33.573235 17142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:34:33.573338 17142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:33.574347 17142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5407985b422a478f94ecec6cb7faac97" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45999 } }
I20260808 04:34:33.574757 17142 leader_election.cc:302] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5407985b422a478f94ecec6cb7faac97; no voters:
I20260808 04:34:33.575133 17142 leader_election.cc:288] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:34:33.575394 17148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:34:33.575656 17148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97 [term 1 LEADER]: Becoming Leader. State: Replica: 5407985b422a478f94ecec6cb7faac97, State: Running, Role: LEADER
I20260808 04:34:33.576004 17148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5407985b422a478f94ecec6cb7faac97" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45999 } }
I20260808 04:34:33.576773 17142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:34:33.578457 17150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5407985b422a478f94ecec6cb7faac97. Latest consensus state: current_term: 1 leader_uuid: "5407985b422a478f94ecec6cb7faac97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5407985b422a478f94ecec6cb7faac97" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45999 } } }
I20260808 04:34:33.578899 17149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5407985b422a478f94ecec6cb7faac97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5407985b422a478f94ecec6cb7faac97" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45999 } } }
I20260808 04:34:33.579085 17150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97 [sys.catalog]: This master's current role is: LEADER
I20260808 04:34:33.579231 17149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97 [sys.catalog]: This master's current role is: LEADER
I20260808 04:34:33.579941 17163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:34:33.583560 17163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:34:33.590900 17163 catalog_manager.cc:1347] Generated new cluster ID: 2179ecf67cbe44deb00497aa5449058f
I20260808 04:34:33.591101 17163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:34:33.607515 17163 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:34:33.608851 17163 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:34:33.619644 17163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5407985b422a478f94ecec6cb7faac97: Generated new TSK 0
I20260808 04:34:33.620473 17163 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
I20260808 04:34:33.664012 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:45999
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:38613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:33.802834 17175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:33.803177 17175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:33.803257 17175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:33.808130 17175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:33.808357 17175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:33.808519 17175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:34:33.815071 17175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:38613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:45999
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:33.816290 17175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:33.817411 17175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:33.830737 17180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:33.830804 17175 server_base.cc:1034] running on GCE node
W20260808 04:34:33.830750 17186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:33.830737 17182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:33.831943 17175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:33.832696 17175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:33.834141 17175 hybrid_clock.cc:648] HybridClock initialized: now 1786163673834041 us; error 114 us; skew 500 ppm
Aug 08 04:34:33 dist-test-slave-drl0 krb5kdc[17016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163673, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:33.837975 17175 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:34:33.839010 17175 webserver.cc:460] Webserver started at http://127.15.228.65:36195/ using document root <none> and password file <none>
I20260808 04:34:33.839627 17175 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:33.839690 17175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:33.839922 17175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:33.842113 17175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "385866a8e4894a10ae4c6581d812f294"
format_stamp: "Formatted at 2026-08-08 04:34:33 on dist-test-slave-drl0"
server_key: "11b5d91f631e6c13f3d32ca6878a1e23"
server_key_iv: "29586c8f0223e7753c14071c3e9caf62"
server_key_version: "encryptionkey@0"
I20260808 04:34:33.842695 17175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "385866a8e4894a10ae4c6581d812f294"
format_stamp: "Formatted at 2026-08-08 04:34:33 on dist-test-slave-drl0"
server_key: "11b5d91f631e6c13f3d32ca6878a1e23"
server_key_iv: "29586c8f0223e7753c14071c3e9caf62"
server_key_version: "encryptionkey@0"
I20260808 04:34:33.847116 17175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:34:33.850608 17194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:33.852406 17175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:34:33.852572 17175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "385866a8e4894a10ae4c6581d812f294"
format_stamp: "Formatted at 2026-08-08 04:34:33 on dist-test-slave-drl0"
server_key: "11b5d91f631e6c13f3d32ca6878a1e23"
server_key_iv: "29586c8f0223e7753c14071c3e9caf62"
server_key_version: "encryptionkey@0"
I20260808 04:34:33.852694 17175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:33.864704 17175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:33.869067 17175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:33.869320 17175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:33.869997 17175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:34:33.870918 17175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:33.871032 17175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:33.871102 17175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:33.871135 17175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:33.882441 17175 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:37775
I20260808 04:34:33.882483 17348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:37775 every 8 connection(s)
I20260808 04:34:33.883495 17175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:34:33.893450 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 17175
I20260808 04:34:33.893684 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
Aug 08 04:34:33 dist-test-slave-drl0 krb5kdc[17016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163673, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:33.894120 16273 external_mini_cluster.cc:1428] Setting key 3b9ff33549344639d9f9068cada03409
I20260808 04:34:33.900264 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:33.885493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:50315 (local address 127.15.228.126:45999)
0808 04:34:33.885676 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:33.885680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:33.886465 (+ 785us) server_negotiation.cc:407] Connection header received
0808 04:34:33.888096 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:33.888099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:33.888149 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:33.888226 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:33.889470 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:33.890026 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:33.891304 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:33.891561 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:33.891778 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:33.895194 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:33.895197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:33.895200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:33.895227 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:33.897137 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:33.897951 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:33.897957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:33.897962 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:33.898062 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:33.898626 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:33.898629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:33.898630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:33.898834 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:33.898916 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:33.900004 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0808 04:34:33.900126 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 04:34:33.901604 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:33.885839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45999 (local address 127.15.228.65:50315)
0808 04:34:33.886306 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:33.886354 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:33.887880 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:33.888424 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:33.888431 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:33.888811 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:33.889295 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:33.889309 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:33.890171 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:33.890174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:33.891183 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:33.891194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:33.891750 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:33.891754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:33.891915 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:33.892506 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:33.892523 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:33.895017 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:33.897356 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:33.897366 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:33.897387 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:33.897790 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:33.898227 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:33.898232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:33.898236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:33.898489 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:33.899029 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:33.899040 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:33.899591 (+ 551us) client_negotiation.cc:769] Sending connection context
0808 04:34:33.899901 (+ 310us) client_negotiation.cc:240] Negotiation successful
0808 04:34:33.900458 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":113,"threads_started":1}
I20260808 04:34:33.903539 17350 heartbeater.cc:347] Connected to a master server at 127.15.228.126:45999
I20260808 04:34:33.903846 17350 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:33.904397 17350 heartbeater.cc:511] Master 127.15.228.126:45999 requested a full tablet report, sending...
I20260808 04:34:33.906210 17084 ts_manager.cc:194] Registered new tserver with Master: 385866a8e4894a10ae4c6581d812f294 (127.15.228.65:37775)
WARNING: no policy specified for kudu/127.15.228.66@KRBTEST.COM; defaulting to no policy
I20260808 04:34:33.907550 17084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:50315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
I20260808 04:34:33.931073 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:45999
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:38613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:34.060858 17356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:34.061154 17356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:34.061203 17356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:34.065230 17356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:34.065289 17356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:34.065445 17356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:34:34.071465 17356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:38613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:45999
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:34.072523 17356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:34.073859 17356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:34.086231 17371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:34.086421 17356 server_base.cc:1034] running on GCE node
W20260808 04:34:34.086220 17368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:34.086220 17369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:34.087211 17356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:34.088385 17356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:34.089711 17356 hybrid_clock.cc:648] HybridClock initialized: now 1786163674089654 us; error 76 us; skew 500 ppm
Aug 08 04:34:34 dist-test-slave-drl0 krb5kdc[17016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163674, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:34.097325 17356 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:34:34.098438 17356 webserver.cc:460] Webserver started at http://127.15.228.66:43743/ using document root <none> and password file <none>
I20260808 04:34:34.099164 17356 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:34.099264 17356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:34.099568 17356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:34.101766 17356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "124992562ca44b5bb93e5e8ceecb74d1"
format_stamp: "Formatted at 2026-08-08 04:34:34 on dist-test-slave-drl0"
server_key: "9b7b11e8ea0366eae25c2ec229b6b255"
server_key_iv: "3e906cf1f289812b92ba6e2880f001ba"
server_key_version: "encryptionkey@0"
I20260808 04:34:34.102378 17356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "124992562ca44b5bb93e5e8ceecb74d1"
format_stamp: "Formatted at 2026-08-08 04:34:34 on dist-test-slave-drl0"
server_key: "9b7b11e8ea0366eae25c2ec229b6b255"
server_key_iv: "3e906cf1f289812b92ba6e2880f001ba"
server_key_version: "encryptionkey@0"
I20260808 04:34:34.106802 17356 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260808 04:34:34.110397 17380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:34.112074 17356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:34:34.112268 17356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "124992562ca44b5bb93e5e8ceecb74d1"
format_stamp: "Formatted at 2026-08-08 04:34:34 on dist-test-slave-drl0"
server_key: "9b7b11e8ea0366eae25c2ec229b6b255"
server_key_iv: "3e906cf1f289812b92ba6e2880f001ba"
server_key_version: "encryptionkey@0"
I20260808 04:34:34.112404 17356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:34.131985 17356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:34.137059 17356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:34.137387 17356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:34.138262 17356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:34:34.140087 17356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:34.140156 17356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:34.140440 17356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:34.140466 17356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:34.150488 17356 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:46681
I20260808 04:34:34.150627 17554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:46681 every 8 connection(s)
I20260808 04:34:34.151571 17356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:34:34.160827 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 17356
I20260808 04:34:34.160995 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:34:34.161278 16273 external_mini_cluster.cc:1428] Setting key b1513bc2c0294cc0c87604e8039c987f
Aug 08 04:34:34 dist-test-slave-drl0 krb5kdc[17016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163674, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:34.169829 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:34.154538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:56703 (local address 127.15.228.126:45999)
0808 04:34:34.154889 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:34.154894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:34.156210 (+ 1316us) server_negotiation.cc:407] Connection header received
0808 04:34:34.157277 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:34.157281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:34.157346 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:34.157476 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:34.158740 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.159344 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.160562 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.160789 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.160914 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:34.164145 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:34.164152 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:34.164159 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:34.164193 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:34.166405 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:34.167165 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:34.167172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:34.167177 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:34.167287 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:34.167938 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:34.167945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:34.167950 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:34.168276 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:34.168407 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:34.169392 (+ 985us) server_negotiation.cc:299] Negotiation successful
0808 04:34:34.169572 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260808 04:34:34.171094 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:34.155107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45999 (local address 127.15.228.66:56703)
0808 04:34:34.156004 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:34.156055 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:34.157047 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:34.157675 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:34.157682 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:34.158059 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:34.158579 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:34.158593 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.159506 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.159509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:34.160421 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:34.160431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.160978 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.160982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:34.161253 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:34.161985 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:34.162010 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:34.163960 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:34.166592 (+ 2632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:34.166602 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:34.166624 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:34.167046 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:34.167486 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:34.167492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:34.167494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:34.167774 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:34.168442 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:34.168495 (+ 53us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:34.168999 (+ 504us) client_negotiation.cc:769] Sending connection context
0808 04:34:34.169182 (+ 183us) client_negotiation.cc:240] Negotiation successful
0808 04:34:34.169683 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":191,"threads_started":1}
I20260808 04:34:34.172803 17555 heartbeater.cc:347] Connected to a master server at 127.15.228.126:45999
WARNING: no policy specified for kudu/127.15.228.67@KRBTEST.COM; defaulting to no policy
I20260808 04:34:34.173225 17555 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:34.174072 17555 heartbeater.cc:511] Master 127.15.228.126:45999 requested a full tablet report, sending...
I20260808 04:34:34.175722 17084 ts_manager.cc:194] Registered new tserver with Master: 124992562ca44b5bb93e5e8ceecb74d1 (127.15.228.66:46681)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.67@KRBTEST.COM" created.
I20260808 04:34:34.176481 17084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:56703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
I20260808 04:34:34.196133 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:45999
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:38613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:34.332127 17566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:34.332459 17566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:34.332540 17566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:34.336798 17566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:34.336889 17566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:34.336987 17566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:34:34.346973 17566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:38613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:45999
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:34.348528 17566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:34.350142 17566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:34.364498 17573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:34.364667 17577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:34.364643 17574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:34.365060 17566 server_base.cc:1034] running on GCE node
I20260808 04:34:34.365653 17566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:34.366601 17566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:34.367831 17566 hybrid_clock.cc:648] HybridClock initialized: now 1786163674367788 us; error 74 us; skew 500 ppm
Aug 08 04:34:34 dist-test-slave-drl0 krb5kdc[17016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163674, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:34.372144 17566 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:34:34.373539 17566 webserver.cc:460] Webserver started at http://127.15.228.67:33775/ using document root <none> and password file <none>
I20260808 04:34:34.374362 17566 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:34.374442 17566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:34.374747 17566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:34.377753 17566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "db379b5a2acc45ec95d1bf5df7765fd0"
format_stamp: "Formatted at 2026-08-08 04:34:34 on dist-test-slave-drl0"
server_key: "0cc291af564a80517b372d724ca2cfa6"
server_key_iv: "eb2cb88ba904b1fc0f7f2d304db850b1"
server_key_version: "encryptionkey@0"
I20260808 04:34:34.378641 17566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "db379b5a2acc45ec95d1bf5df7765fd0"
format_stamp: "Formatted at 2026-08-08 04:34:34 on dist-test-slave-drl0"
server_key: "0cc291af564a80517b372d724ca2cfa6"
server_key_iv: "eb2cb88ba904b1fc0f7f2d304db850b1"
server_key_version: "encryptionkey@0"
I20260808 04:34:34.384493 17566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 04:34:34.388721 17588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:34.390725 17566 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:34:34.390990 17566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "db379b5a2acc45ec95d1bf5df7765fd0"
format_stamp: "Formatted at 2026-08-08 04:34:34 on dist-test-slave-drl0"
server_key: "0cc291af564a80517b372d724ca2cfa6"
server_key_iv: "eb2cb88ba904b1fc0f7f2d304db850b1"
server_key_version: "encryptionkey@0"
I20260808 04:34:34.391153 17566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:34.410744 17566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:34.414359 17566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:34.414577 17566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:34.415285 17566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:34:34.416515 17566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:34.416594 17566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:34.416669 17566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:34.416712 17566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:34.431131 17566 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:34055
I20260808 04:34:34.431195 17747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:34055 every 8 connection(s)
I20260808 04:34:34.432130 17566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:34:34.436524 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 17566
I20260808 04:34:34.436659 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:34:34.436955 16273 external_mini_cluster.cc:1428] Setting key 26e8bb857c60aa7b511d07586688e58c
Aug 08 04:34:34 dist-test-slave-drl0 krb5kdc[17016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163674, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:34.447829 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:34.435038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:58725 (local address 127.15.228.126:45999)
0808 04:34:34.435200 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:34.435204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:34.436310 (+ 1106us) server_negotiation.cc:407] Connection header received
0808 04:34:34.437338 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:34.437341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:34.437398 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:34.437490 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:34.438816 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.439432 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.440705 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.440933 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.441058 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:34.444031 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:34.444035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:34.444037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:34.444060 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:34.445502 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:34.446029 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:34.446032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:34.446034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:34.446123 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:34.446693 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:34.446697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:34.446698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:34.446903 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:34.447069 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:34.447599 (+ 530us) server_negotiation.cc:299] Negotiation successful
0808 04:34:34.447701 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:34:34.448830 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:34.435432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45999 (local address 127.15.228.67:58725)
0808 04:34:34.436135 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:34.436181 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:34.437127 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:34.437681 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:34.437687 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:34.438050 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:34.438658 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:34.438672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.439570 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.439573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:34.440560 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:34.440570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.441103 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.441107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:34.441279 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:34.441915 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:34.441933 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:34.443848 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:34.445632 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:34.445637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:34.445649 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:34.445909 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:34.446250 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:34.446253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:34.446254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:34.446426 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:34.447077 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:34.447083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:34.447344 (+ 261us) client_negotiation.cc:769] Sending connection context
0808 04:34:34.447521 (+ 177us) client_negotiation.cc:240] Negotiation successful
0808 04:34:34.447816 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":229,"threads_started":1}
I20260808 04:34:34.450850 17748 heartbeater.cc:347] Connected to a master server at 127.15.228.126:45999
I20260808 04:34:34.451354 17748 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:34.452219 17748 heartbeater.cc:511] Master 127.15.228.126:45999 requested a full tablet report, sending...
I20260808 04:34:34.454321 17086 ts_manager.cc:194] Registered new tserver with Master: db379b5a2acc45ec95d1bf5df7765fd0 (127.15.228.67:34055)
I20260808 04:34:34.455268 17086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:58725
I20260808 04:34:34.463392 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:34:34.477550 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:34.468186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44090 (local address 127.15.228.126:45999)
0808 04:34:34.468473 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:34.468479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:34.468683 (+ 204us) server_negotiation.cc:407] Connection header received
0808 04:34:34.468863 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:34.468866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:34.468923 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:34.469019 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:34.469834 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.470385 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.471293 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.471520 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.471637 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:34.473646 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:34.473650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:34.473653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:34.473692 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:34.475351 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:34.475870 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:34.475880 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:34.475882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:34.475964 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:34.476506 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:34.476510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:34.476512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:34.476806 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:34.476971 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:34.477218 (+ 247us) server_negotiation.cc:299] Negotiation successful
0808 04:34:34.477348 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260808 04:34:34.484405 17086 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:34:34.488013 17086 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:34:34.507387 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:34.500562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:37775 (local address 127.0.0.1:51516)
0808 04:34:34.501398 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:34.501411 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:34.501488 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:34.502210 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:34.502214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:34.502233 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:34.502445 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:34.502454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.503892 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.503895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:34.505151 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:34.505162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.506559 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.506563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:34.506748 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:34.506814 (+ 66us) client_negotiation.cc:769] Sending connection context
0808 04:34:34.506978 (+ 164us) client_negotiation.cc:240] Negotiation successful
0808 04:34:34.507167 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":36,"thread_start_us":302,"threads_started":1}
I20260808 04:34:34.507755 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:34.500731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.15.228.66:46681)
0808 04:34:34.501618 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:34.501637 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:34.501685 (+ 48us) server_negotiation.cc:407] Connection header received
0808 04:34:34.501813 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:34.501822 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:34.502163 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:34.502573 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:34.502913 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.504351 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.505764 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.506592 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.506704 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:34.506964 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:34.507401 (+ 437us) server_negotiation.cc:299] Negotiation successful
0808 04:34:34.507506 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":120,"threads_started":1}
I20260808 04:34:34.508087 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:34.499843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:34055 (local address 127.0.0.1:51546)
0808 04:34:34.500873 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:34.500977 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:34.501132 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:34.501979 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:34.501988 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:34.502038 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:34.502297 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:34.502310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.504554 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.504558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:34.505771 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:34.505786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.507310 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.507314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:34.507413 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:34.507449 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 04:34:34.507527 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 04:34:34.507788 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":5,"thread_start_us":67,"threads_started":1}
I20260808 04:34:34.508231 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:34.500027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51546 (local address 127.15.228.67:34055)
0808 04:34:34.500387 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:34.500400 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:34.501072 (+ 672us) server_negotiation.cc:407] Connection header received
0808 04:34:34.501311 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:34.501317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:34.501539 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:34.501726 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:34.502592 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.504283 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.505970 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.506883 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.507070 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:34.507251 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:34.507907 (+ 656us) server_negotiation.cc:299] Negotiation successful
0808 04:34:34.508061 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":117,"threads_started":1}
I20260808 04:34:34.508466 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:34.500486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:46681 (local address 127.0.0.1:46038)
0808 04:34:34.500958 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:34.500977 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:34.501117 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:34.502448 (+ 1331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:34.502451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:34.502469 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:34.502592 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:34.502599 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.504519 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.504523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:34.505627 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:34.505635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.507034 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.507040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:34.507207 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:34.507248 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 04:34:34.507356 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 04:34:34.507453 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":1,"thread_start_us":201,"threads_started":1}
I20260808 04:34:34.508708 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:34.501110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51516 (local address 127.15.228.65:37775)
0808 04:34:34.501815 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:34.501821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:34.501843 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:34:34.501917 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:34.501921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:34.502059 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:34.502209 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:34.502588 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.503746 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.505333 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.506376 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.508150 (+ 1774us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:34.508321 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:34.508382 (+ 61us) server_negotiation.cc:299] Negotiation successful
0808 04:34:34.508495 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":264,"threads_started":1}
I20260808 04:34:34.511782 17654 tablet_service.cc:1467] Processing CreateTablet for tablet 94e4509e46c54d5eb32a78ece851f9c3 (DEFAULT_TABLE table=test-table [id=08683bc3df024f30b25f824a5411ea14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:34.512742 17255 tablet_service.cc:1467] Processing CreateTablet for tablet 94e4509e46c54d5eb32a78ece851f9c3 (DEFAULT_TABLE table=test-table [id=08683bc3df024f30b25f824a5411ea14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:34.513203 17654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e4509e46c54d5eb32a78ece851f9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:34.513717 17255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e4509e46c54d5eb32a78ece851f9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:34.517900 17458 tablet_service.cc:1467] Processing CreateTablet for tablet 94e4509e46c54d5eb32a78ece851f9c3 (DEFAULT_TABLE table=test-table [id=08683bc3df024f30b25f824a5411ea14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:34.519222 17458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e4509e46c54d5eb32a78ece851f9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:34.523919 17774 tablet_bootstrap.cc:492] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0: Bootstrap starting.
I20260808 04:34:34.525218 17775 tablet_bootstrap.cc:492] T 94e4509e46c54d5eb32a78ece851f9c3 P 385866a8e4894a10ae4c6581d812f294: Bootstrap starting.
I20260808 04:34:34.527624 17775 tablet_bootstrap.cc:654] T 94e4509e46c54d5eb32a78ece851f9c3 P 385866a8e4894a10ae4c6581d812f294: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:34.527887 17774 tablet_bootstrap.cc:654] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:34.528788 17775 log.cc:824] T 94e4509e46c54d5eb32a78ece851f9c3 P 385866a8e4894a10ae4c6581d812f294: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:34.530014 17774 log.cc:824] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:34.530601 17777 tablet_bootstrap.cc:492] T 94e4509e46c54d5eb32a78ece851f9c3 P 124992562ca44b5bb93e5e8ceecb74d1: Bootstrap starting.
I20260808 04:34:34.532335 17774 tablet_bootstrap.cc:492] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0: No bootstrap required, opened a new log
I20260808 04:34:34.532578 17774 ts_tablet_manager.cc:1397] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260808 04:34:34.532703 17775 tablet_bootstrap.cc:492] T 94e4509e46c54d5eb32a78ece851f9c3 P 385866a8e4894a10ae4c6581d812f294: No bootstrap required, opened a new log
I20260808 04:34:34.532888 17775 ts_tablet_manager.cc:1397] T 94e4509e46c54d5eb32a78ece851f9c3 P 385866a8e4894a10ae4c6581d812f294: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 04:34:34.534099 17777 tablet_bootstrap.cc:654] T 94e4509e46c54d5eb32a78ece851f9c3 P 124992562ca44b5bb93e5e8ceecb74d1: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:34.535355 17777 log.cc:824] T 94e4509e46c54d5eb32a78ece851f9c3 P 124992562ca44b5bb93e5e8ceecb74d1: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:34.536852 17774 raft_consensus.cc:348] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db379b5a2acc45ec95d1bf5df7765fd0" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34055 } } peers { permanent_uuid: "385866a8e4894a10ae4c6581d812f294" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 37775 } } peers { permanent_uuid: "124992562ca44b5bb93e5e8ceecb74d1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 46681 } }
I20260808 04:34:34.537190 17774 raft_consensus.cc:374] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:34.537256 17774 raft_consensus.cc:729] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db379b5a2acc45ec95d1bf5df7765fd0, State: Initialized, Role: FOLLOWER
I20260808 04:34:34.537725 17774 consensus_queue.cc:260] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db379b5a2acc45ec95d1bf5df7765fd0" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34055 } } peers { permanent_uuid: "385866a8e4894a10ae4c6581d812f294" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 37775 } } peers { permanent_uuid: "124992562ca44b5bb93e5e8ceecb74d1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 46681 } }
I20260808 04:34:34.538534 17748 heartbeater.cc:503] Master 127.15.228.126:45999 was elected leader, sending a full tablet report...
I20260808 04:34:34.538587 17777 tablet_bootstrap.cc:492] T 94e4509e46c54d5eb32a78ece851f9c3 P 124992562ca44b5bb93e5e8ceecb74d1: No bootstrap required, opened a new log
I20260808 04:34:34.538892 17774 ts_tablet_manager.cc:1428] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 04:34:34.538898 17777 ts_tablet_manager.cc:1397] T 94e4509e46c54d5eb32a78ece851f9c3 P 124992562ca44b5bb93e5e8ceecb74d1: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260808 04:34:34.540575 17775 raft_consensus.cc:348] T 94e4509e46c54d5eb32a78ece851f9c3 P 385866a8e4894a10ae4c6581d812f294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db379b5a2acc45ec95d1bf5df7765fd0" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34055 } } peers { permanent_uuid: "385866a8e4894a10ae4c6581d812f294" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 37775 } } peers { permanent_uuid: "124992562ca44b5bb93e5e8ceecb74d1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 46681 } }
I20260808 04:34:34.540920 17775 raft_consensus.cc:374] T 94e4509e46c54d5eb32a78ece851f9c3 P 385866a8e4894a10ae4c6581d812f294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:34.541002 17775 raft_consensus.cc:729] T 94e4509e46c54d5eb32a78ece851f9c3 P 385866a8e4894a10ae4c6581d812f294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 385866a8e4894a10ae4c6581d812f294, State: Initialized, Role: FOLLOWER
I20260808 04:34:34.541563 17775 consensus_queue.cc:260] T 94e4509e46c54d5eb32a78ece851f9c3 P 385866a8e4894a10ae4c6581d812f294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db379b5a2acc45ec95d1bf5df7765fd0" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34055 } } peers { permanent_uuid: "385866a8e4894a10ae4c6581d812f294" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 37775 } } peers { permanent_uuid: "124992562ca44b5bb93e5e8ceecb74d1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 46681 } }
I20260808 04:34:34.542543 17350 heartbeater.cc:503] Master 127.15.228.126:45999 was elected leader, sending a full tablet report...
I20260808 04:34:34.543201 17775 ts_tablet_manager.cc:1428] T 94e4509e46c54d5eb32a78ece851f9c3 P 385866a8e4894a10ae4c6581d812f294: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260808 04:34:34.545892 17777 raft_consensus.cc:348] T 94e4509e46c54d5eb32a78ece851f9c3 P 124992562ca44b5bb93e5e8ceecb74d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db379b5a2acc45ec95d1bf5df7765fd0" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34055 } } peers { permanent_uuid: "385866a8e4894a10ae4c6581d812f294" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 37775 } } peers { permanent_uuid: "124992562ca44b5bb93e5e8ceecb74d1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 46681 } }
I20260808 04:34:34.546271 17777 raft_consensus.cc:374] T 94e4509e46c54d5eb32a78ece851f9c3 P 124992562ca44b5bb93e5e8ceecb74d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:34.546449 17777 raft_consensus.cc:729] T 94e4509e46c54d5eb32a78ece851f9c3 P 124992562ca44b5bb93e5e8ceecb74d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124992562ca44b5bb93e5e8ceecb74d1, State: Initialized, Role: FOLLOWER
I20260808 04:34:34.547330 17777 consensus_queue.cc:260] T 94e4509e46c54d5eb32a78ece851f9c3 P 124992562ca44b5bb93e5e8ceecb74d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db379b5a2acc45ec95d1bf5df7765fd0" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34055 } } peers { permanent_uuid: "385866a8e4894a10ae4c6581d812f294" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 37775 } } peers { permanent_uuid: "124992562ca44b5bb93e5e8ceecb74d1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 46681 } }
I20260808 04:34:34.548640 17555 heartbeater.cc:503] Master 127.15.228.126:45999 was elected leader, sending a full tablet report...
I20260808 04:34:34.548827 17777 ts_tablet_manager.cc:1428] T 94e4509e46c54d5eb32a78ece851f9c3 P 124992562ca44b5bb93e5e8ceecb74d1: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
W20260808 04:34:34.635412 17351 tablet.cc:2369] T 94e4509e46c54d5eb32a78ece851f9c3 P 385866a8e4894a10ae4c6581d812f294: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:34:34.649624 17780 raft_consensus.cc:482] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:34:34.649865 17780 raft_consensus.cc:504] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db379b5a2acc45ec95d1bf5df7765fd0" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34055 } } peers { permanent_uuid: "385866a8e4894a10ae4c6581d812f294" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 37775 } } peers { permanent_uuid: "124992562ca44b5bb93e5e8ceecb74d1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 46681 } }
I20260808 04:34:34.651801 17780 leader_election.cc:288] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 385866a8e4894a10ae4c6581d812f294 (127.15.228.65:37775), 124992562ca44b5bb93e5e8ceecb74d1 (127.15.228.66:46681)
W20260808 04:34:34.653131 17556 tablet.cc:2369] T 94e4509e46c54d5eb32a78ece851f9c3 P 124992562ca44b5bb93e5e8ceecb74d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:34:34.656028 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:34.651778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:37775 (local address 127.15.228.67:39551)
0808 04:34:34.651961 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:34.651979 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:34.652095 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:34.652568 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:34.652571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:34.652601 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:34.652752 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:34.652766 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.653696 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.653699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:34.654741 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:34.654750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.655598 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.655602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:34.655710 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:34.655738 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:34:34.655822 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 04:34:34.655890 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260808 04:34:34.656203 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:34.652013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:39551 (local address 127.15.228.65:37775)
0808 04:34:34.652298 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:34.652303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:34.652316 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:34:34.652374 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:34.652376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:34.652417 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:34.652520 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:34.652886 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.653560 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.654896 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.655474 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.655579 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:34.655623 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:34.655900 (+ 277us) server_negotiation.cc:299] Negotiation successful
0808 04:34:34.656057 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260808 04:34:34.656945 17283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e4509e46c54d5eb32a78ece851f9c3" candidate_uuid: "db379b5a2acc45ec95d1bf5df7765fd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "385866a8e4894a10ae4c6581d812f294" is_pre_election: true
I20260808 04:34:34.657698 17283 raft_consensus.cc:2454] T 94e4509e46c54d5eb32a78ece851f9c3 P 385866a8e4894a10ae4c6581d812f294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db379b5a2acc45ec95d1bf5df7765fd0 in term 0.
I20260808 04:34:34.657783 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:34.652568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:46129 (local address 127.15.228.66:46681)
0808 04:34:34.652718 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:34.652722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:34.652923 (+ 201us) server_negotiation.cc:407] Connection header received
0808 04:34:34.653533 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:34.653537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:34.653593 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:34.653667 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:34.654118 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.654728 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.656227 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.657010 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.657170 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:34.657267 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:34.657588 (+ 321us) server_negotiation.cc:299] Negotiation successful
0808 04:34:34.657664 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:34:34.657841 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:34.652431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:46681 (local address 127.15.228.67:46129)
0808 04:34:34.652787 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:34.652798 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:34.652933 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:34.653768 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:34.653771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:34.653789 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:34.653987 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:34.653994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.654837 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.654844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:34.656079 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:34.656089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.657239 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.657244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:34.657415 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:34.657444 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 04:34:34.657557 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 04:34:34.657636 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":157,"threads_started":1}
I20260808 04:34:34.658432 17591 leader_election.cc:302] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 385866a8e4894a10ae4c6581d812f294, db379b5a2acc45ec95d1bf5df7765fd0; no voters:
I20260808 04:34:34.658766 17780 raft_consensus.cc:2781] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:34:34.658731 17486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e4509e46c54d5eb32a78ece851f9c3" candidate_uuid: "db379b5a2acc45ec95d1bf5df7765fd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124992562ca44b5bb93e5e8ceecb74d1" is_pre_election: true
I20260808 04:34:34.658881 17780 raft_consensus.cc:482] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:34:34.658969 17780 raft_consensus.cc:3037] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:34.659094 17486 raft_consensus.cc:2454] T 94e4509e46c54d5eb32a78ece851f9c3 P 124992562ca44b5bb93e5e8ceecb74d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db379b5a2acc45ec95d1bf5df7765fd0 in term 0.
I20260808 04:34:34.660533 17780 raft_consensus.cc:504] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db379b5a2acc45ec95d1bf5df7765fd0" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34055 } } peers { permanent_uuid: "385866a8e4894a10ae4c6581d812f294" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 37775 } } peers { permanent_uuid: "124992562ca44b5bb93e5e8ceecb74d1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 46681 } }
I20260808 04:34:34.661049 17780 leader_election.cc:288] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0 [CANDIDATE]: Term 1 election: Requested vote from peers 385866a8e4894a10ae4c6581d812f294 (127.15.228.65:37775), 124992562ca44b5bb93e5e8ceecb74d1 (127.15.228.66:46681)
I20260808 04:34:34.661495 17283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e4509e46c54d5eb32a78ece851f9c3" candidate_uuid: "db379b5a2acc45ec95d1bf5df7765fd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "385866a8e4894a10ae4c6581d812f294"
I20260808 04:34:34.661482 17486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e4509e46c54d5eb32a78ece851f9c3" candidate_uuid: "db379b5a2acc45ec95d1bf5df7765fd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124992562ca44b5bb93e5e8ceecb74d1"
I20260808 04:34:34.661608 17283 raft_consensus.cc:3037] T 94e4509e46c54d5eb32a78ece851f9c3 P 385866a8e4894a10ae4c6581d812f294 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:34.661769 17486 raft_consensus.cc:3037] T 94e4509e46c54d5eb32a78ece851f9c3 P 124992562ca44b5bb93e5e8ceecb74d1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:34.663039 17283 raft_consensus.cc:2454] T 94e4509e46c54d5eb32a78ece851f9c3 P 385866a8e4894a10ae4c6581d812f294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db379b5a2acc45ec95d1bf5df7765fd0 in term 1.
I20260808 04:34:34.663383 17591 leader_election.cc:302] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 385866a8e4894a10ae4c6581d812f294, db379b5a2acc45ec95d1bf5df7765fd0; no voters:
I20260808 04:34:34.663519 17486 raft_consensus.cc:2454] T 94e4509e46c54d5eb32a78ece851f9c3 P 124992562ca44b5bb93e5e8ceecb74d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db379b5a2acc45ec95d1bf5df7765fd0 in term 1.
I20260808 04:34:34.663627 17780 raft_consensus.cc:2781] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:34:34.663878 17780 raft_consensus.cc:686] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0 [term 1 LEADER]: Becoming Leader. State: Replica: db379b5a2acc45ec95d1bf5df7765fd0, State: Running, Role: LEADER
I20260808 04:34:34.664572 17780 consensus_queue.cc:237] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db379b5a2acc45ec95d1bf5df7765fd0" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34055 } } peers { permanent_uuid: "385866a8e4894a10ae4c6581d812f294" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 37775 } } peers { permanent_uuid: "124992562ca44b5bb93e5e8ceecb74d1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 46681 } }
I20260808 04:34:34.668710 17084 catalog_manager.cc:5495] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to db379b5a2acc45ec95d1bf5df7765fd0 (127.15.228.67). New cstate: current_term: 1 leader_uuid: "db379b5a2acc45ec95d1bf5df7765fd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db379b5a2acc45ec95d1bf5df7765fd0" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "385866a8e4894a10ae4c6581d812f294" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 37775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124992562ca44b5bb93e5e8ceecb74d1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 46681 } health_report { overall_health: UNKNOWN } } }
W20260808 04:34:34.684062 17749 tablet.cc:2369] T 94e4509e46c54d5eb32a78ece851f9c3 P db379b5a2acc45ec95d1bf5df7765fd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 04:34:34 dist-test-slave-drl0 krb5kdc[17016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.65@KRBTEST.COM
I20260808 04:34:34.706362 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:34.693800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51530 (local address 127.15.228.65:37775)
0808 04:34:34.694021 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:34.694025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:34.694174 (+ 149us) server_negotiation.cc:407] Connection header received
0808 04:34:34.694282 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:34.694285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:34.694331 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:34.694402 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:34.695724 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.696452 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.697640 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.697913 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.698025 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:34.701319 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:34.701333 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:34.701352 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:34.701397 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:34.704032 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:34.704741 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:34.704747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:34.704749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:34.704842 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:34.705204 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:34.705207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:34.705208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:34.705513 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:34.705675 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:34.705905 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 04:34:34.706055 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
Aug 08 04:34:34 dist-test-slave-drl0 krb5kdc[17016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163674, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 04:34:34 dist-test-slave-drl0 krb5kdc[17016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163674, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.228.65@KRBTEST.COM
I20260808 04:34:34.741578 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:34.731100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51542 (local address 127.15.228.65:37775)
0808 04:34:34.731315 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:34.731319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:34.731473 (+ 154us) server_negotiation.cc:407] Connection header received
0808 04:34:34.731616 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:34.731618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:34.731674 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:34.731760 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:34.732634 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.733312 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.734308 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.734614 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.734711 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:34.737867 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:34.737877 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:34.737880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:34.737905 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:34.739591 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:34.740050 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:34.740054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:34.740056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:34.740143 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:34.740578 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:34.740581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:34.740583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:34.740766 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:34.740913 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:34.741270 (+ 357us) server_negotiation.cc:299] Negotiation successful
0808 04:34:34.741409 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
Aug 08 04:34:34 dist-test-slave-drl0 krb5kdc[17016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 04:34:34 dist-test-slave-drl0 krb5kdc[17016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.65@KRBTEST.COM
I20260808 04:34:34.776896 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:34.764814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51544 (local address 127.15.228.65:37775)
0808 04:34:34.765053 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:34.765059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:34.765388 (+ 329us) server_negotiation.cc:407] Connection header received
0808 04:34:34.765530 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:34.765533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:34.765604 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:34.765691 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:34.766567 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.767379 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.768319 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:34.768550 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:34.768645 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:34.771742 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:34.771746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:34.771748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:34.771777 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:34.773990 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:34.774779 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:34.774784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:34.774785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:34.774904 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:34.775449 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:34.775452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:34.775454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:34.775652 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:34.775894 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:34.776348 (+ 454us) server_negotiation.cc:299] Negotiation successful
0808 04:34:34.776491 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260808 04:34:34.779103 17230 tablet_service.cc:2908] Rejecting scan request for tablet 94e4509e46c54d5eb32a78ece851f9c3: Uninitialized: clean time has not yet been initialized
I20260808 04:34:34.782755 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 17175
I20260808 04:34:34.800866 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 17356
I20260808 04:34:34.821276 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 17566
I20260808 04:34:34.831733 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 17034
2026-08-08T04:34:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2911 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:34:34 dist-test-slave-drl0 krb5kdc[17805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:34:34 dist-test-slave-drl0 krb5kdc[17805](info): set up 2 sockets
Aug 08 04:34:34 dist-test-slave-drl0 krb5kdc[17805](info): commencing operation
krb5kdc: starting...
W20260808 04:34:36.093891 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:34:36 dist-test-slave-drl0 krb5kdc[17805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:34:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:34:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:34:36.235071 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:43123
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:40743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:43123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:36.360504 17823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:36.360982 17823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:36.361073 17823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:36.366756 17823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:34:36.366892 17823 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 04:34:36.366926 17823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:36.366983 17823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:36.367044 17823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:34:36.367085 17823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:34:36.373461 17823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:40743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:43123
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:43123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:36.374650 17823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:36.375747 17823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:36.385589 17830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:36.385579 17829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:36.385879 17823 server_base.cc:1034] running on GCE node
W20260808 04:34:36.385958 17832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:36.386457 17823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:36.387676 17823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:36.388895 17823 hybrid_clock.cc:648] HybridClock initialized: now 1786163676388888 us; error 44 us; skew 500 ppm
Aug 08 04:34:36 dist-test-slave-drl0 krb5kdc[17805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163676, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:36.391904 17823 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:34:36.392879 17823 webserver.cc:460] Webserver started at http://127.15.228.126:40535/ using document root <none> and password file <none>
I20260808 04:34:36.393461 17823 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:36.393553 17823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:36.393803 17823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:36.396155 17823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "d74e5ff2e7a3460a9fcbab63bfd9fdcd"
format_stamp: "Formatted at 2026-08-08 04:34:36 on dist-test-slave-drl0"
server_key: "c666e51ce2158a57ed1a5c4497b7f5cd"
server_key_iv: "86971953312cd589583778cafeed2baa"
server_key_version: "encryptionkey@0"
I20260808 04:34:36.396768 17823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "d74e5ff2e7a3460a9fcbab63bfd9fdcd"
format_stamp: "Formatted at 2026-08-08 04:34:36 on dist-test-slave-drl0"
server_key: "c666e51ce2158a57ed1a5c4497b7f5cd"
server_key_iv: "86971953312cd589583778cafeed2baa"
server_key_version: "encryptionkey@0"
I20260808 04:34:36.401475 17823 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:34:36.404503 17841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:36.406344 17823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:34:36.406620 17823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "d74e5ff2e7a3460a9fcbab63bfd9fdcd"
format_stamp: "Formatted at 2026-08-08 04:34:36 on dist-test-slave-drl0"
server_key: "c666e51ce2158a57ed1a5c4497b7f5cd"
server_key_iv: "86971953312cd589583778cafeed2baa"
server_key_version: "encryptionkey@0"
I20260808 04:34:36.406795 17823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:36.419834 17823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:36.423883 17823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:36.424103 17823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:36.434674 17823 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:43123
I20260808 04:34:36.434695 17929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:43123 every 8 connection(s)
I20260808 04:34:36.436241 17823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:34:36.439580 17930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:36.443845 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 17823
I20260808 04:34:36.444013 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:34:36.444340 16273 external_mini_cluster.cc:1428] Setting key ec4ccf36c83fa07dc730766ebd9ddfe7
I20260808 04:34:36.447397 17930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd: Bootstrap starting.
I20260808 04:34:36.450176 17930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:36.451354 17930 log.cc:824] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:36.454838 17930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd: No bootstrap required, opened a new log
Aug 08 04:34:36 dist-test-slave-drl0 krb5kdc[17805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:36.459853 17930 raft_consensus.cc:348] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d74e5ff2e7a3460a9fcbab63bfd9fdcd" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43123 } }
I20260808 04:34:36.460181 17930 raft_consensus.cc:374] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:36.460247 17930 raft_consensus.cc:729] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74e5ff2e7a3460a9fcbab63bfd9fdcd, State: Initialized, Role: FOLLOWER
I20260808 04:34:36.460806 17930 consensus_queue.cc:260] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d74e5ff2e7a3460a9fcbab63bfd9fdcd" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43123 } }
I20260808 04:34:36.460999 17930 raft_consensus.cc:388] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:34:36.461092 17930 raft_consensus.cc:482] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:34:36.461225 17930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:36.463317 17930 raft_consensus.cc:504] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d74e5ff2e7a3460a9fcbab63bfd9fdcd" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43123 } }
I20260808 04:34:36.463783 17930 leader_election.cc:302] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d74e5ff2e7a3460a9fcbab63bfd9fdcd; no voters:
I20260808 04:34:36.464185 17930 leader_election.cc:288] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:34:36.464344 17936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:34:36.464582 17936 raft_consensus.cc:686] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd [term 1 LEADER]: Becoming Leader. State: Replica: d74e5ff2e7a3460a9fcbab63bfd9fdcd, State: Running, Role: LEADER
I20260808 04:34:36.464973 17936 consensus_queue.cc:237] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d74e5ff2e7a3460a9fcbab63bfd9fdcd" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43123 } }
I20260808 04:34:36.465397 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:36.446480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51140 (local address 127.15.228.126:43123)
0808 04:34:36.447317 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:36.447339 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:36.447389 (+ 50us) server_negotiation.cc:407] Connection header received
0808 04:34:36.448385 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:36.448409 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:36.448879 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:36.449359 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:36.450549 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:36.452124 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:36.453405 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:36.454281 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:36.454495 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:36.458248 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:36.458256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:36.458276 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:36.458316 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:36.460746 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:36.461462 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:36.461471 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:36.461480 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:36.461633 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:36.462076 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:36.462082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:36.462084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:36.463102 (+ 1018us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:36.463475 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:36.463997 (+ 522us) server_negotiation.cc:299] Negotiation successful
0808 04:34:36.464370 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":240,"threads_started":1}
I20260808 04:34:36.465595 17930 sys_catalog.cc:564] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:34:36.466711 17940 sys_catalog.cc:455] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d74e5ff2e7a3460a9fcbab63bfd9fdcd. Latest consensus state: current_term: 1 leader_uuid: "d74e5ff2e7a3460a9fcbab63bfd9fdcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d74e5ff2e7a3460a9fcbab63bfd9fdcd" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43123 } } }
I20260808 04:34:36.466845 17940 sys_catalog.cc:458] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd [sys.catalog]: This master's current role is: LEADER
I20260808 04:34:36.468374 17937 sys_catalog.cc:455] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d74e5ff2e7a3460a9fcbab63bfd9fdcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d74e5ff2e7a3460a9fcbab63bfd9fdcd" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43123 } } }
I20260808 04:34:36.468478 17937 sys_catalog.cc:458] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd [sys.catalog]: This master's current role is: LEADER
I20260808 04:34:36.468664 17956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:34:36.471706 17956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:34:36.477476 17956 catalog_manager.cc:1347] Generated new cluster ID: bac21d105e7045acb112988620462071
I20260808 04:34:36.477556 17956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:34:36.545154 17956 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:34:36.546249 17956 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:34:36.551013 17956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd: Generated new TSK 0
I20260808 04:34:36.551725 17956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
I20260808 04:34:36.592007 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:43123
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:40743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:36.717064 17973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:36.717342 17973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:36.717388 17973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:36.721558 17973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:36.721617 17973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:36.721657 17973 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:34:36.721688 17973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:34:36.727169 17973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:40743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.228.126:43123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:36.728256 17973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:36.729346 17973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:36.740796 17986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:36.740818 17982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:36.740806 17980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:36.741214 17973 server_base.cc:1034] running on GCE node
I20260808 04:34:36.741662 17973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:36.742316 17973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:36.743690 17973 hybrid_clock.cc:648] HybridClock initialized: now 1786163676743617 us; error 85 us; skew 500 ppm
Aug 08 04:34:36 dist-test-slave-drl0 krb5kdc[17805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163676, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:36.747057 17973 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:34:36.748004 17973 webserver.cc:460] Webserver started at http://127.15.228.65:41061/ using document root <none> and password file <none>
I20260808 04:34:36.748591 17973 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:36.748683 17973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:36.748921 17973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:36.751065 17973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "57e705f331044f1faf9f93230e88ac35"
format_stamp: "Formatted at 2026-08-08 04:34:36 on dist-test-slave-drl0"
server_key: "eabd532dfc5e49a9dc92c8faa20e7519"
server_key_iv: "d0e374bff502cff66e4e3ce667a0e79c"
server_key_version: "encryptionkey@0"
I20260808 04:34:36.751703 17973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "57e705f331044f1faf9f93230e88ac35"
format_stamp: "Formatted at 2026-08-08 04:34:36 on dist-test-slave-drl0"
server_key: "eabd532dfc5e49a9dc92c8faa20e7519"
server_key_iv: "d0e374bff502cff66e4e3ce667a0e79c"
server_key_version: "encryptionkey@0"
I20260808 04:34:36.756601 17973 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260808 04:34:36.759723 17994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:36.761710 17973 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:34:36.761893 17973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "57e705f331044f1faf9f93230e88ac35"
format_stamp: "Formatted at 2026-08-08 04:34:36 on dist-test-slave-drl0"
server_key: "eabd532dfc5e49a9dc92c8faa20e7519"
server_key_iv: "d0e374bff502cff66e4e3ce667a0e79c"
server_key_version: "encryptionkey@0"
I20260808 04:34:36.762210 17973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:36.790025 17973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:36.794849 17973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:36.795104 17973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:36.797161 17973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:36.797232 17973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:36.797307 17973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:36.797335 17973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:34:36 dist-test-slave-drl0 krb5kdc[17805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163676, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:36.808578 17973 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:45189
I20260808 04:34:36.808641 18158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:45189 every 8 connection(s)
I20260808 04:34:36.809477 17973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:34:36.812750 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 17973
I20260808 04:34:36.812886 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:34:36.813172 16273 external_mini_cluster.cc:1428] Setting key c0977907d6746383f6b8e2d088245f33
I20260808 04:34:36.813845 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:36.798017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:37911 (local address 127.15.228.126:43123)
0808 04:34:36.798194 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:36.798201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:36.799634 (+ 1433us) server_negotiation.cc:407] Connection header received
0808 04:34:36.801076 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:36.801079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:36.801130 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:36.801234 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:36.803055 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:36.803800 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:36.805611 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:36.805858 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:36.805955 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:36.808969 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:36.808972 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:36.808975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:36.809010 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:36.810650 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:36.811448 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:36.811451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:36.811452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:36.811532 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:36.811864 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:36.811866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:36.811867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:36.812122 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:36.812213 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:36.813397 (+ 1184us) server_negotiation.cc:299] Negotiation successful
0808 04:34:36.813610 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 04:34:36.814920 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:36.798876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:43123 (local address 127.15.228.65:37911)
0808 04:34:36.799440 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:36.799504 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:36.800807 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:36.801443 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:36.801453 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:36.802002 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:36.802851 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:36.802880 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:36.803916 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:36.803920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:36.805447 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:36.805458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:36.806034 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:36.806038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:36.806265 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:36.807046 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:36.807063 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:36.808782 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:36.810785 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:36.810794 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:36.810806 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:36.811277 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:36.811620 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:36.811622 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:36.811623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:36.811769 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:36.812483 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:36.812494 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:36.812971 (+ 477us) client_negotiation.cc:769] Sending connection context
0808 04:34:36.813291 (+ 320us) client_negotiation.cc:240] Negotiation successful
0808 04:34:36.813768 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":120,"threads_started":1}
I20260808 04:34:36.816730 18159 heartbeater.cc:347] Connected to a master server at 127.15.228.126:43123
I20260808 04:34:36.816992 18159 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:36.817643 18159 heartbeater.cc:511] Master 127.15.228.126:43123 requested a full tablet report, sending...
I20260808 04:34:36.819293 17864 ts_manager.cc:194] Registered new tserver with Master: 57e705f331044f1faf9f93230e88ac35 (127.15.228.65:45189)
I20260808 04:34:36.820770 17864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:37911
WARNING: no policy specified for kudu/127.15.228.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.66@KRBTEST.COM" created.
I20260808 04:34:36.830297 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:36.819952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:40817 (local address 127.15.228.126:43123)
0808 04:34:36.820072 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:36.820077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:36.820547 (+ 470us) server_negotiation.cc:407] Connection header received
0808 04:34:36.820679 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:36.820683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:36.820729 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:36.820819 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:36.821448 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:36.822222 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:36.823878 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:36.824422 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:36.824571 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:36.826069 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:36.826073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:36.826076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:36.826104 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:36.828114 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:36.828718 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:36.828721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:36.828722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:36.828801 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:36.829241 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:36.829247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:36.829249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:36.829516 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:36.829637 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:36.830055 (+ 418us) server_negotiation.cc:299] Negotiation successful
0808 04:34:36.830164 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:34:36.830657 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:36.819887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:43123 (local address 127.15.228.65:40817)
0808 04:34:36.820439 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:36.820458 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:36.820575 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:36.820858 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:36.820888 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:36.821147 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:36.821301 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:36.821310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:36.822395 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:36.822399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:36.823585 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:36.823597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:36.824649 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:36.824654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:36.824778 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:36.825545 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:36.825557 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:36.825912 (+ 355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:36.828254 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:36.828257 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:36.828259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:36.828582 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:36.828924 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:36.828928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:36.828932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:36.829039 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:36.829698 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:36.829702 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:36.829830 (+ 128us) client_negotiation.cc:769] Sending connection context
0808 04:34:36.829978 (+ 148us) client_negotiation.cc:240] Negotiation successful
0808 04:34:36.830189 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
I20260808 04:34:36.848048 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:43123
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:40743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:36.984951 18173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:36.985301 18173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:36.985388 18173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:36.990387 18173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:36.990483 18173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:36.990556 18173 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:34:36.990621 18173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:34:36.996621 18173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:40743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.228.126:43123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:36.997922 18173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:36.999111 18173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:37.009680 18180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:37.009685 18179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:37.010044 18173 server_base.cc:1034] running on GCE node
W20260808 04:34:37.009663 18182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:37.011636 18173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:37.012342 18173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:37.013551 18173 hybrid_clock.cc:648] HybridClock initialized: now 1786163677013498 us; error 64 us; skew 500 ppm
Aug 08 04:34:37 dist-test-slave-drl0 krb5kdc[17805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163677, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:37.016633 18173 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:34:37.017613 18173 webserver.cc:460] Webserver started at http://127.15.228.66:34341/ using document root <none> and password file <none>
I20260808 04:34:37.018332 18173 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:37.018591 18173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:37.019161 18173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:37.021389 18173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "2a92ec8c7f354da8937e305553b4553c"
format_stamp: "Formatted at 2026-08-08 04:34:37 on dist-test-slave-drl0"
server_key: "d7f1b40ddf3949a46024e09de4a5cac7"
server_key_iv: "73c48a6828a52e23e3eb72d3c8780dbd"
server_key_version: "encryptionkey@0"
I20260808 04:34:37.022056 18173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "2a92ec8c7f354da8937e305553b4553c"
format_stamp: "Formatted at 2026-08-08 04:34:37 on dist-test-slave-drl0"
server_key: "d7f1b40ddf3949a46024e09de4a5cac7"
server_key_iv: "73c48a6828a52e23e3eb72d3c8780dbd"
server_key_version: "encryptionkey@0"
I20260808 04:34:37.026175 18173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260808 04:34:37.028995 18193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:37.030392 18173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 04:34:37.030562 18173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "2a92ec8c7f354da8937e305553b4553c"
format_stamp: "Formatted at 2026-08-08 04:34:37 on dist-test-slave-drl0"
server_key: "d7f1b40ddf3949a46024e09de4a5cac7"
server_key_iv: "73c48a6828a52e23e3eb72d3c8780dbd"
server_key_version: "encryptionkey@0"
I20260808 04:34:37.030691 18173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:37.046471 18173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:37.050040 18173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:37.050285 18173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:37.051970 18173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:37.052042 18173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:37.052115 18173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:37.052158 18173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:34:37 dist-test-slave-drl0 krb5kdc[17805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163677, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:37.065843 18173 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:39509
I20260808 04:34:37.065987 18369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:39509 every 8 connection(s)
I20260808 04:34:37.066991 18173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:34:37.068979 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.053065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:49225 (local address 127.15.228.126:43123)
0808 04:34:37.053278 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:37.053283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:37.054565 (+ 1282us) server_negotiation.cc:407] Connection header received
0808 04:34:37.055602 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:37.055606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:37.055672 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:37.055777 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:37.057000 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.057547 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.059046 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.059364 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.059478 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.063377 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:37.063381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:37.063384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:37.063419 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:37.065968 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:37.066993 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:37.066998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:37.067000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:37.067124 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:37.067602 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:37.067606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:37.067608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:37.067904 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:37.068042 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:37.068707 (+ 665us) server_negotiation.cc:299] Negotiation successful
0808 04:34:37.068844 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 04:34:37.069723 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 18173
I20260808 04:34:37.069828 18212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.053812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:43123 (local address 127.15.228.66:49225)
0808 04:34:37.054371 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:37.054416 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:37.055387 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:37.055956 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:37.055966 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:37.056354 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:37.056846 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.056860 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.057687 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.057690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.058857 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.058868 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.059526 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.059530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.059742 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.060488 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:37.060514 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:37.063194 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:37.066294 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:37.066307 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:37.066332 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:37.066800 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:37.067226 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:37.067229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:37.067232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:37.067412 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:37.068063 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:37.068070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:37.068385 (+ 315us) client_negotiation.cc:769] Sending connection context
0808 04:34:37.068559 (+ 174us) client_negotiation.cc:240] Negotiation successful
0808 04:34:37.068910 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":143,"threads_started":1}
I20260808 04:34:37.069821 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:34:37.070174 16273 external_mini_cluster.cc:1428] Setting key fddb9e27f513638e4a0ecab7ce8fe0ed
I20260808 04:34:37.071168 18370 heartbeater.cc:347] Connected to a master server at 127.15.228.126:43123
I20260808 04:34:37.071308 18370 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:37.071830 18370 heartbeater.cc:511] Master 127.15.228.126:43123 requested a full tablet report, sending...
I20260808 04:34:37.072893 17863 ts_manager.cc:194] Registered new tserver with Master: 2a92ec8c7f354da8937e305553b4553c (127.15.228.66:39509)
I20260808 04:34:37.073443 17863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:49225
WARNING: no policy specified for kudu/127.15.228.67@KRBTEST.COM; defaulting to no policy
I20260808 04:34:37.084213 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.074700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:43123 (local address 127.15.228.66:49293)
0808 04:34:37.075006 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:37.075023 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:37.075125 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:37.075400 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:37.075403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:37.075604 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:37.075713 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.075719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.076575 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.076581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.077621 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.077634 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.078433 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.078439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.078561 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.079233 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:37.079254 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:37.079798 (+ 544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:37.082220 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:37.082226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:37.082230 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:37.082507 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:37.082995 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:37.082999 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:37.083001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:37.083113 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:37.083723 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:37.083725 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:37.083829 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 04:34:37.083923 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 04:34:37.084047 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":92,"threads_started":1}
I20260808 04:34:37.084604 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.074693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:49293 (local address 127.15.228.126:43123)
0808 04:34:37.074839 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:37.074844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:37.075109 (+ 265us) server_negotiation.cc:407] Connection header received
0808 04:34:37.075213 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:37.075218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:37.075272 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:37.075442 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:37.075832 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.076430 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.077789 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.078164 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.078250 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.080103 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:37.080108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:37.080111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:37.080155 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:37.082024 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:37.082650 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:37.082655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:37.082657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:37.082765 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:37.083265 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:37.083269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:37.083270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:37.083588 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:37.083696 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:37.084181 (+ 485us) server_negotiation.cc:299] Negotiation successful
0808 04:34:37.084428 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
I20260808 04:34:37.104249 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:43123
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:40743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:37.236054 18385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:37.236332 18385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:37.236414 18385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:37.242358 18385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:37.242443 18385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:37.242487 18385 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:34:37.242566 18385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:34:37.249640 18385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:40743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.228.126:43123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:37.251184 18385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:37.252460 18385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:37.263736 18397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:37.263710 18394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:37.263736 18392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:37.264091 18385 server_base.cc:1034] running on GCE node
I20260808 04:34:37.264571 18385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:37.265223 18385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:37.266433 18385 hybrid_clock.cc:648] HybridClock initialized: now 1786163677266348 us; error 94 us; skew 500 ppm
Aug 08 04:34:37 dist-test-slave-drl0 krb5kdc[17805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163677, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:37.270103 18385 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:34:37.271062 18385 webserver.cc:460] Webserver started at http://127.15.228.67:37435/ using document root <none> and password file <none>
I20260808 04:34:37.271713 18385 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:37.271780 18385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:37.272010 18385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:37.274183 18385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "0528dba3541b426cacdbf67126d9851d"
format_stamp: "Formatted at 2026-08-08 04:34:37 on dist-test-slave-drl0"
server_key: "fd14f00a03fb12c54d815d18eb3e46a7"
server_key_iv: "e04fa2af088dbe87d65913959ee9d635"
server_key_version: "encryptionkey@0"
I20260808 04:34:37.274837 18385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "0528dba3541b426cacdbf67126d9851d"
format_stamp: "Formatted at 2026-08-08 04:34:37 on dist-test-slave-drl0"
server_key: "fd14f00a03fb12c54d815d18eb3e46a7"
server_key_iv: "e04fa2af088dbe87d65913959ee9d635"
server_key_version: "encryptionkey@0"
I20260808 04:34:37.280027 18385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 04:34:37.283586 18412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:37.285043 18385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:34:37.285215 18385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "0528dba3541b426cacdbf67126d9851d"
format_stamp: "Formatted at 2026-08-08 04:34:37 on dist-test-slave-drl0"
server_key: "fd14f00a03fb12c54d815d18eb3e46a7"
server_key_iv: "e04fa2af088dbe87d65913959ee9d635"
server_key_version: "encryptionkey@0"
I20260808 04:34:37.285375 18385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:37.312641 18385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:37.316148 18385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:37.316358 18385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:37.319084 18385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:37.319221 18385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:37.319305 18385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:37.319347 18385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:34:37 dist-test-slave-drl0 krb5kdc[17805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163677, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:37.334663 18385 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:39899
I20260808 04:34:37.334801 18585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:39899 every 8 connection(s)
I20260808 04:34:37.335783 18385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:34:37.335881 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.319448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:51117 (local address 127.15.228.126:43123)
0808 04:34:37.319704 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:37.319711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:37.320603 (+ 892us) server_negotiation.cc:407] Connection header received
0808 04:34:37.321867 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:37.321872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:37.321956 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:37.322137 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:37.324192 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.325216 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.326622 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.326887 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.327087 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.330981 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:37.330985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:37.330988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:37.331022 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:37.333087 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:37.333791 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:37.333795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:37.333797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:37.333918 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:37.334440 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:37.334446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:37.334449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:37.334747 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:37.334878 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:37.335541 (+ 663us) server_negotiation.cc:299] Negotiation successful
0808 04:34:37.335685 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 04:34:37.336995 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.319876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:43123 (local address 127.15.228.67:51117)
0808 04:34:37.320399 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:37.320453 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:37.321595 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:37.322539 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:37.322547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:37.323127 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:37.323934 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.323951 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.325414 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.325418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.326454 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.326465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.327133 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.327137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.327328 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.328026 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:37.328047 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:37.330693 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:37.333220 (+ 2527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:37.333227 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:37.333242 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:37.333640 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:37.334052 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:37.334058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:37.334063 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:37.334283 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:37.334880 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:37.334888 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:37.335268 (+ 380us) client_negotiation.cc:769] Sending connection context
0808 04:34:37.335495 (+ 227us) client_negotiation.cc:240] Negotiation successful
0808 04:34:37.335886 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":155,"threads_started":1}
I20260808 04:34:37.338311 18586 heartbeater.cc:347] Connected to a master server at 127.15.228.126:43123
I20260808 04:34:37.338454 18586 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:37.339072 18586 heartbeater.cc:511] Master 127.15.228.126:43123 requested a full tablet report, sending...
I20260808 04:34:37.340602 17864 ts_manager.cc:194] Registered new tserver with Master: 0528dba3541b426cacdbf67126d9851d (127.15.228.67:39899)
I20260808 04:34:37.341758 17864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:51117
I20260808 04:34:37.344722 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 18385
I20260808 04:34:37.344898 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:34:37.345327 16273 external_mini_cluster.cc:1428] Setting key d73eda2029d138ef67ab7732c1146c8d
I20260808 04:34:37.348516 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:34:37.350818 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.340979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:43123 (local address 127.15.228.67:38469)
0808 04:34:37.341500 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:37.341523 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:37.341729 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:37.342104 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:37.342107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:37.342441 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:37.342614 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.342638 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.343650 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.343655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.344312 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.344351 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.345029 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.345034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.345160 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.346087 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:37.346102 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:37.346583 (+ 481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:37.348820 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:37.348823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:37.348825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:37.349117 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:37.349485 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:37.349488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:37.349489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:37.349585 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:37.350190 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:37.350194 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:37.350331 (+ 137us) client_negotiation.cc:769] Sending connection context
0808 04:34:37.350454 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 04:34:37.350635 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":105,"threads_started":1}
I20260808 04:34:37.351007 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.341106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:38469 (local address 127.15.228.126:43123)
0808 04:34:37.341428 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:37.341431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:37.341714 (+ 283us) server_negotiation.cc:407] Connection header received
0808 04:34:37.341845 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:37.341848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:37.341906 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:37.342048 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:37.342786 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.343492 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.344565 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.344819 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.344981 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.346793 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:37.346798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:37.346801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:37.346845 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:37.348667 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:37.349227 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:37.349233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:37.349235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:37.349333 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:37.349691 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:37.349696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:37.349701 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:37.349994 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:37.350106 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:37.350549 (+ 443us) server_negotiation.cc:299] Negotiation successful
0808 04:34:37.350701 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
Aug 08 04:34:37 dist-test-slave-drl0 krb5kdc[17805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163677, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 04:34:37 dist-test-slave-drl0 krb5kdc[17805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163677, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:37.380172 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.370875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51156 (local address 127.15.228.126:43123)
0808 04:34:37.371060 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:37.371064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:37.371077 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:34:37.371118 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:37.371120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:37.371160 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:37.371256 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:37.372017 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.372553 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.373392 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.373611 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.373724 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.376407 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:37.376411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:37.376413 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:37.376435 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:37.378096 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:37.378629 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:37.378632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:37.378633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:37.378712 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:37.379299 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:37.379302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:37.379303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:37.379501 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:37.379618 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:37.379943 (+ 325us) server_negotiation.cc:299] Negotiation successful
0808 04:34:37.380049 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 04:34:37.383410 17864 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:34:37.386662 17864 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:34:37.402704 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.396259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:39509 (local address 127.0.0.1:43286)
0808 04:34:37.397531 (+ 1272us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:37.397544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:37.397614 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:37.398501 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:37.398504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:37.398529 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:37.398646 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.398653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.399959 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.399962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.400929 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.400939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.402098 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.402103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.402232 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.402275 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 04:34:37.402398 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 04:34:37.402529 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1215,"mutex_wait_us":39,"spinlock_wait_cycles":8960,"thread_start_us":117,"threads_started":1}
I20260808 04:34:37.402725 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.396989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43286 (local address 127.15.228.66:39509)
0808 04:34:37.398041 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:37.398054 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:37.398077 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:34:37.398152 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:37.398156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:37.398344 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:37.398497 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:37.398772 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.399834 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.401076 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.401940 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.402075 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.402254 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:37.402445 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 04:34:37.402548 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":128,"threads_started":1}
I20260808 04:34:37.403064 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.396556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48700 (local address 127.15.228.67:39899)
0808 04:34:37.397030 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:37.397042 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:37.397237 (+ 195us) server_negotiation.cc:407] Connection header received
0808 04:34:37.397461 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:37.397466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:37.397681 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:37.397858 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:37.398168 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.399390 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.401262 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.402013 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.402104 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.402254 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:37.402759 (+ 505us) server_negotiation.cc:299] Negotiation successful
0808 04:34:37.402867 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":109,"threads_started":1}
I20260808 04:34:37.403461 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.396257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:39899 (local address 127.0.0.1:48700)
0808 04:34:37.397094 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:37.397133 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:37.397306 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:37.397857 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:37.397863 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:37.397885 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:37.398035 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.398043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.399528 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.399531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.401115 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.401128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.402331 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.402335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.402523 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.402595 (+ 72us) client_negotiation.cc:769] Sending connection context
0808 04:34:37.402667 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 04:34:37.403297 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":684,"thread_start_us":87,"threads_started":1}
I20260808 04:34:37.404323 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.397367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37058 (local address 127.15.228.65:45189)
0808 04:34:37.397826 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:37.397832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:37.398095 (+ 263us) server_negotiation.cc:407] Connection header received
0808 04:34:37.398290 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:37.398300 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:37.398475 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:37.398624 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:37.399037 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.399958 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.401437 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.402031 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.402152 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.402309 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:37.404070 (+ 1761us) server_negotiation.cc:299] Negotiation successful
0808 04:34:37.404186 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":99,"threads_started":1}
I20260808 04:34:37.404582 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.396259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:45189 (local address 127.0.0.1:37058)
0808 04:34:37.397964 (+ 1705us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:37.397977 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:37.398127 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:37.398791 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:37.398793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:37.398818 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:37.398881 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.398885 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.400120 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.400124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.401315 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.401323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.402331 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.402335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.402519 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.402594 (+ 75us) client_negotiation.cc:769] Sending connection context
0808 04:34:37.402667 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 04:34:37.402760 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1648,"mutex_wait_us":291,"thread_start_us":118,"threads_started":1}
I20260808 04:34:37.405972 18499 tablet_service.cc:1467] Processing CreateTablet for tablet f074b7c72ff14af08c128a8abb5f5818 (DEFAULT_TABLE table=test-table [id=cddfef186d2b4984956a9b9d513b8120]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:37.405987 18276 tablet_service.cc:1467] Processing CreateTablet for tablet f074b7c72ff14af08c128a8abb5f5818 (DEFAULT_TABLE table=test-table [id=cddfef186d2b4984956a9b9d513b8120]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:37.407047 18499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f074b7c72ff14af08c128a8abb5f5818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:37.407045 18276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f074b7c72ff14af08c128a8abb5f5818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:37.409762 18062 tablet_service.cc:1467] Processing CreateTablet for tablet f074b7c72ff14af08c128a8abb5f5818 (DEFAULT_TABLE table=test-table [id=cddfef186d2b4984956a9b9d513b8120]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:37.410903 18062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f074b7c72ff14af08c128a8abb5f5818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:37.414604 18618 tablet_bootstrap.cc:492] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c: Bootstrap starting.
I20260808 04:34:37.416545 18621 tablet_bootstrap.cc:492] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d: Bootstrap starting.
I20260808 04:34:37.417124 18618 tablet_bootstrap.cc:654] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:37.417905 18618 log.cc:824] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:37.418656 18621 tablet_bootstrap.cc:654] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:37.418690 18622 tablet_bootstrap.cc:492] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35: Bootstrap starting.
I20260808 04:34:37.419514 18621 log.cc:824] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:37.420989 18622 tablet_bootstrap.cc:654] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:37.421262 18621 tablet_bootstrap.cc:492] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d: No bootstrap required, opened a new log
I20260808 04:34:37.421351 18618 tablet_bootstrap.cc:492] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c: No bootstrap required, opened a new log
I20260808 04:34:37.421501 18621 ts_tablet_manager.cc:1397] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260808 04:34:37.421536 18618 ts_tablet_manager.cc:1397] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 04:34:37.421798 18622 log.cc:824] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:37.423776 18622 tablet_bootstrap.cc:492] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35: No bootstrap required, opened a new log
I20260808 04:34:37.423961 18622 ts_tablet_manager.cc:1397] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:34:37.424908 18621 raft_consensus.cc:348] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.425163 18621 raft_consensus.cc:374] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:37.425218 18621 raft_consensus.cc:729] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0528dba3541b426cacdbf67126d9851d, State: Initialized, Role: FOLLOWER
I20260808 04:34:37.425683 18621 consensus_queue.cc:260] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.426770 18621 ts_tablet_manager.cc:1428] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260808 04:34:37.426410 18618 raft_consensus.cc:348] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.426923 18618 raft_consensus.cc:374] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:37.427170 18618 raft_consensus.cc:729] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a92ec8c7f354da8937e305553b4553c, State: Initialized, Role: FOLLOWER
I20260808 04:34:37.427204 18586 heartbeater.cc:503] Master 127.15.228.126:43123 was elected leader, sending a full tablet report...
I20260808 04:34:37.427627 18618 consensus_queue.cc:260] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.428319 18370 heartbeater.cc:503] Master 127.15.228.126:43123 was elected leader, sending a full tablet report...
I20260808 04:34:37.428601 18618 ts_tablet_manager.cc:1428] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 04:34:37.432358 18622 raft_consensus.cc:348] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.432827 18622 raft_consensus.cc:374] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:37.432900 18622 raft_consensus.cc:729] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e705f331044f1faf9f93230e88ac35, State: Initialized, Role: FOLLOWER
I20260808 04:34:37.433455 18622 consensus_queue.cc:260] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.434218 18159 heartbeater.cc:503] Master 127.15.228.126:43123 was elected leader, sending a full tablet report...
I20260808 04:34:37.438973 18622 ts_tablet_manager.cc:1428] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35: Time spent starting tablet: real 0.015s user 0.009s sys 0.000s
I20260808 04:34:37.539630 18627 raft_consensus.cc:482] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:34:37.540037 18627 raft_consensus.cc:504] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.541788 18627 leader_election.cc:288] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57e705f331044f1faf9f93230e88ac35 (127.15.228.65:45189), 0528dba3541b426cacdbf67126d9851d (127.15.228.67:39899)
I20260808 04:34:37.548017 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.542269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:45359 (local address 127.15.228.65:45189)
0808 04:34:37.542452 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:37.542459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:37.542693 (+ 234us) server_negotiation.cc:407] Connection header received
0808 04:34:37.542766 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:37.542768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:37.542824 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:37.542981 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:37.543432 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.544241 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.546724 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.547324 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.547490 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.547553 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:37.547787 (+ 234us) server_negotiation.cc:299] Negotiation successful
0808 04:34:37.547873 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 04:34:37.548029 18212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.542134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:45189 (local address 127.15.228.66:45359)
0808 04:34:37.542451 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:37.542469 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:37.542646 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:37.543021 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:37.543023 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:37.543037 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:37.543259 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.543270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.544481 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.544485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.546512 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.546530 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.547511 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.547515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.547621 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.547646 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:34:37.547738 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 04:34:37.547823 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":188}
I20260808 04:34:37.548540 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.542363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:33661 (local address 127.15.228.67:39899)
0808 04:34:37.542524 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:37.542528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:37.542881 (+ 353us) server_negotiation.cc:407] Connection header received
0808 04:34:37.542972 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:37.542975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:37.543021 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:37.543106 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:37.543432 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.544241 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.547249 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.547834 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.547967 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.548011 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:37.548311 (+ 300us) server_negotiation.cc:299] Negotiation successful
0808 04:34:37.548415 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:34:37.548791 18092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f074b7c72ff14af08c128a8abb5f5818" candidate_uuid: "2a92ec8c7f354da8937e305553b4553c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e705f331044f1faf9f93230e88ac35" is_pre_election: true
I20260808 04:34:37.548931 18629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.542263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:39899 (local address 127.15.228.66:33661)
0808 04:34:37.542748 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:37.542762 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:37.542864 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:37.543136 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:37.543139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:37.543151 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:37.543305 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.543315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.544481 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.544485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.547110 (+ 2625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.547122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.547995 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.547999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.548108 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.548159 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 04:34:37.548695 (+ 536us) client_negotiation.cc:240] Negotiation successful
0808 04:34:37.548780 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":190,"thread_start_us":172,"threads_started":1}
I20260808 04:34:37.549218 18092 raft_consensus.cc:2454] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a92ec8c7f354da8937e305553b4553c in term 0.
I20260808 04:34:37.549605 18528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f074b7c72ff14af08c128a8abb5f5818" candidate_uuid: "2a92ec8c7f354da8937e305553b4553c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0528dba3541b426cacdbf67126d9851d" is_pre_election: true
I20260808 04:34:37.549832 18201 leader_election.cc:302] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a92ec8c7f354da8937e305553b4553c, 57e705f331044f1faf9f93230e88ac35; no voters:
I20260808 04:34:37.549939 18528 raft_consensus.cc:2454] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a92ec8c7f354da8937e305553b4553c in term 0.
I20260808 04:34:37.550181 18627 raft_consensus.cc:2781] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:34:37.550272 18627 raft_consensus.cc:482] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:34:37.550325 18627 raft_consensus.cc:3037] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:37.551698 18627 raft_consensus.cc:504] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.552103 18627 leader_election.cc:288] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [CANDIDATE]: Term 1 election: Requested vote from peers 57e705f331044f1faf9f93230e88ac35 (127.15.228.65:45189), 0528dba3541b426cacdbf67126d9851d (127.15.228.67:39899)
I20260808 04:34:37.552536 18528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f074b7c72ff14af08c128a8abb5f5818" candidate_uuid: "2a92ec8c7f354da8937e305553b4553c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0528dba3541b426cacdbf67126d9851d"
I20260808 04:34:37.552688 18528 raft_consensus.cc:3037] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:37.554189 18528 raft_consensus.cc:2454] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a92ec8c7f354da8937e305553b4553c in term 1.
I20260808 04:34:37.554497 18092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f074b7c72ff14af08c128a8abb5f5818" candidate_uuid: "2a92ec8c7f354da8937e305553b4553c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e705f331044f1faf9f93230e88ac35"
I20260808 04:34:37.554652 18092 raft_consensus.cc:3037] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:37.554687 18202 leader_election.cc:302] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0528dba3541b426cacdbf67126d9851d, 2a92ec8c7f354da8937e305553b4553c; no voters:
I20260808 04:34:37.554883 18627 raft_consensus.cc:2781] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:34:37.555174 18627 raft_consensus.cc:686] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [term 1 LEADER]: Becoming Leader. State: Replica: 2a92ec8c7f354da8937e305553b4553c, State: Running, Role: LEADER
I20260808 04:34:37.555553 18627 consensus_queue.cc:237] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.556064 18092 raft_consensus.cc:2454] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a92ec8c7f354da8937e305553b4553c in term 1.
I20260808 04:34:37.559430 17864 catalog_manager.cc:5495] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a92ec8c7f354da8937e305553b4553c (127.15.228.66). New cstate: current_term: 1 leader_uuid: "2a92ec8c7f354da8937e305553b4553c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } health_report { overall_health: UNKNOWN } } }
W20260808 04:34:37.561040 18160 tablet.cc:2369] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:34:37.568562 18374 tablet.cc:2369] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:34:37.587174 18587 tablet.cc:2369] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:34:37.604935 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.600385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.15.228.66:39509)
0808 04:34:37.600586 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:37.600590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:37.600636 (+ 46us) server_negotiation.cc:407] Connection header received
0808 04:34:37.601004 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:37.601008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:37.601082 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:37.601201 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:34:37.601584 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.602127 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.603154 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.603382 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.603476 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.603866 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:34:37.604359 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:34:37.604464 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:37.604648 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 04:34:37.604741 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 04:34:37.612701 18092 raft_consensus.cc:1261] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35 [term 1 FOLLOWER]: Refusing update from remote peer 2a92ec8c7f354da8937e305553b4553c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:34:37.612702 18528 raft_consensus.cc:1261] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d [term 1 FOLLOWER]: Refusing update from remote peer 2a92ec8c7f354da8937e305553b4553c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:34:37.613917 18630 consensus_queue.cc:1035] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [LEADER]: Connected to new peer: Peer: permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:34:37.616304 18630 consensus_queue.cc:1035] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:34:37.638111 17864 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:34:37.638442 17864 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:34:37.641953 17864 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd: Sending DeleteTablet for 3 replicas of tablet f074b7c72ff14af08c128a8abb5f5818
I20260808 04:34:37.642984 18062 tablet_service.cc:1514] Processing DeleteTablet for tablet f074b7c72ff14af08c128a8abb5f5818 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:37 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:37058
I20260808 04:34:37.643013 18276 tablet_service.cc:1514] Processing DeleteTablet for tablet f074b7c72ff14af08c128a8abb5f5818 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:37 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:43286
I20260808 04:34:37.643306 18499 tablet_service.cc:1514] Processing DeleteTablet for tablet f074b7c72ff14af08c128a8abb5f5818 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:37 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:48700
I20260808 04:34:37.643716 18647 tablet_replica.cc:330] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35: stopping tablet replica
I20260808 04:34:37.643913 18648 tablet_replica.cc:330] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c: stopping tablet replica
I20260808 04:34:37.644222 18648 raft_consensus.cc:2229] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:34:37.644217 18647 raft_consensus.cc:2229] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:34:37.644464 17864 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 04:34:37.644611 18648 raft_consensus.cc:2258] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 04:34:37.644902 17864 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:34:37.644903 18647 raft_consensus.cc:2258] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:34:37.645437 18649 tablet_replica.cc:330] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d: stopping tablet replica
I20260808 04:34:37.646457 18649 raft_consensus.cc:2229] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:34:37.646994 18649 raft_consensus.cc:2258] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:34:37.646807 18647 ts_tablet_manager.cc:1905] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:34:37.648698 18648 ts_tablet_manager.cc:1905] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:34:37.651657 18499 tablet_service.cc:1467] Processing CreateTablet for tablet 70df860e03ed4caab4b7dfa4aa735e57 (DEFAULT_TABLE table=test-table [id=9f44b0f5741b4eeb9912bb2c7f0d1a1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:37.651680 18062 tablet_service.cc:1467] Processing CreateTablet for tablet 70df860e03ed4caab4b7dfa4aa735e57 (DEFAULT_TABLE table=test-table [id=9f44b0f5741b4eeb9912bb2c7f0d1a1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:37.651938 18499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70df860e03ed4caab4b7dfa4aa735e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:37.651969 18062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70df860e03ed4caab4b7dfa4aa735e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:37.653339 18648 ts_tablet_manager.cc:1918] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:34:37.653507 18648 log.cc:1196] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/f074b7c72ff14af08c128a8abb5f5818
I20260808 04:34:37.653921 18648 ts_tablet_manager.cc:1939] T f074b7c72ff14af08c128a8abb5f5818 P 2a92ec8c7f354da8937e305553b4553c: Deleting consensus metadata
I20260808 04:34:37.654390 18276 tablet_service.cc:1467] Processing CreateTablet for tablet 70df860e03ed4caab4b7dfa4aa735e57 (DEFAULT_TABLE table=test-table [id=9f44b0f5741b4eeb9912bb2c7f0d1a1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:37.654657 18276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70df860e03ed4caab4b7dfa4aa735e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:37.656229 17846 catalog_manager.cc:4841] TS 2a92ec8c7f354da8937e305553b4553c (127.15.228.66:39509): tablet f074b7c72ff14af08c128a8abb5f5818 (table test-table [id=cddfef186d2b4984956a9b9d513b8120]) successfully deleted
I20260808 04:34:37.656697 18621 tablet_bootstrap.cc:492] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d: Bootstrap starting.
I20260808 04:34:37.657132 18647 ts_tablet_manager.cc:1918] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:34:37.657315 18647 log.cc:1196] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-0/wal/wals/f074b7c72ff14af08c128a8abb5f5818
I20260808 04:34:37.657810 18647 ts_tablet_manager.cc:1939] T f074b7c72ff14af08c128a8abb5f5818 P 57e705f331044f1faf9f93230e88ac35: Deleting consensus metadata
I20260808 04:34:37.658270 18621 tablet_bootstrap.cc:654] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:37.658684 18618 tablet_bootstrap.cc:492] T 70df860e03ed4caab4b7dfa4aa735e57 P 2a92ec8c7f354da8937e305553b4553c: Bootstrap starting.
I20260808 04:34:37.659879 18618 tablet_bootstrap.cc:654] T 70df860e03ed4caab4b7dfa4aa735e57 P 2a92ec8c7f354da8937e305553b4553c: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:37.660187 18621 tablet_bootstrap.cc:492] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d: No bootstrap required, opened a new log
I20260808 04:34:37.660274 18621 ts_tablet_manager.cc:1397] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 04:34:37.660720 17849 catalog_manager.cc:4841] TS 57e705f331044f1faf9f93230e88ac35 (127.15.228.65:45189): tablet f074b7c72ff14af08c128a8abb5f5818 (table test-table [id=cddfef186d2b4984956a9b9d513b8120]) successfully deleted
I20260808 04:34:37.660894 18621 raft_consensus.cc:348] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.661051 18621 raft_consensus.cc:374] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:37.661101 18621 raft_consensus.cc:729] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0528dba3541b426cacdbf67126d9851d, State: Initialized, Role: FOLLOWER
I20260808 04:34:37.661232 18621 consensus_queue.cc:260] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.661433 18622 tablet_bootstrap.cc:492] T 70df860e03ed4caab4b7dfa4aa735e57 P 57e705f331044f1faf9f93230e88ac35: Bootstrap starting.
I20260808 04:34:37.661541 18621 ts_tablet_manager.cc:1428] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260808 04:34:37.662602 18622 tablet_bootstrap.cc:654] T 70df860e03ed4caab4b7dfa4aa735e57 P 57e705f331044f1faf9f93230e88ac35: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:37.664105 18622 tablet_bootstrap.cc:492] T 70df860e03ed4caab4b7dfa4aa735e57 P 57e705f331044f1faf9f93230e88ac35: No bootstrap required, opened a new log
I20260808 04:34:37.664217 18622 ts_tablet_manager.cc:1397] T 70df860e03ed4caab4b7dfa4aa735e57 P 57e705f331044f1faf9f93230e88ac35: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:34:37.665001 18622 raft_consensus.cc:348] T 70df860e03ed4caab4b7dfa4aa735e57 P 57e705f331044f1faf9f93230e88ac35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.665129 18622 raft_consensus.cc:374] T 70df860e03ed4caab4b7dfa4aa735e57 P 57e705f331044f1faf9f93230e88ac35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:37.665172 18622 raft_consensus.cc:729] T 70df860e03ed4caab4b7dfa4aa735e57 P 57e705f331044f1faf9f93230e88ac35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e705f331044f1faf9f93230e88ac35, State: Initialized, Role: FOLLOWER
I20260808 04:34:37.665355 18622 consensus_queue.cc:260] T 70df860e03ed4caab4b7dfa4aa735e57 P 57e705f331044f1faf9f93230e88ac35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.665706 18622 ts_tablet_manager.cc:1428] T 70df860e03ed4caab4b7dfa4aa735e57 P 57e705f331044f1faf9f93230e88ac35: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260808 04:34:37.666584 18618 tablet_bootstrap.cc:492] T 70df860e03ed4caab4b7dfa4aa735e57 P 2a92ec8c7f354da8937e305553b4553c: No bootstrap required, opened a new log
I20260808 04:34:37.666673 18618 ts_tablet_manager.cc:1397] T 70df860e03ed4caab4b7dfa4aa735e57 P 2a92ec8c7f354da8937e305553b4553c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260808 04:34:37.667043 18649 ts_tablet_manager.cc:1905] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:34:37.667447 18618 raft_consensus.cc:348] T 70df860e03ed4caab4b7dfa4aa735e57 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.667567 18618 raft_consensus.cc:374] T 70df860e03ed4caab4b7dfa4aa735e57 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:37.667608 18618 raft_consensus.cc:729] T 70df860e03ed4caab4b7dfa4aa735e57 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a92ec8c7f354da8937e305553b4553c, State: Initialized, Role: FOLLOWER
I20260808 04:34:37.667728 18618 consensus_queue.cc:260] T 70df860e03ed4caab4b7dfa4aa735e57 P 2a92ec8c7f354da8937e305553b4553c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.668005 18618 ts_tablet_manager.cc:1428] T 70df860e03ed4caab4b7dfa4aa735e57 P 2a92ec8c7f354da8937e305553b4553c: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260808 04:34:37.670336 18649 ts_tablet_manager.cc:1918] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:34:37.670459 18649 log.cc:1196] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/f074b7c72ff14af08c128a8abb5f5818
I20260808 04:34:37.670768 18649 ts_tablet_manager.cc:1939] T f074b7c72ff14af08c128a8abb5f5818 P 0528dba3541b426cacdbf67126d9851d: Deleting consensus metadata
I20260808 04:34:37.672101 17850 catalog_manager.cc:4841] TS 0528dba3541b426cacdbf67126d9851d (127.15.228.67:39899): tablet f074b7c72ff14af08c128a8abb5f5818 (table test-table [id=cddfef186d2b4984956a9b9d513b8120]) successfully deleted
I20260808 04:34:37.675662 18626 raft_consensus.cc:482] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:34:37.675763 18626 raft_consensus.cc:504] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.677431 18626 leader_election.cc:288] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57e705f331044f1faf9f93230e88ac35 (127.15.228.65:45189), 2a92ec8c7f354da8937e305553b4553c (127.15.228.66:39509)
I20260808 04:34:37.682597 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.677600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:45189 (local address 127.15.228.67:57707)
0808 04:34:37.677947 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:37.677964 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:37.678054 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:37.679015 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:37.679018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:37.679036 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:37.679193 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.679200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.680308 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.680311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.681211 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.681221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.682029 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.682033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.682249 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.682269 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 04:34:37.682365 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 04:34:37.682445 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":60}
I20260808 04:34:37.683059 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.678473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:57707 (local address 127.15.228.65:45189)
0808 04:34:37.678650 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:37.678656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:37.678671 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:34:37.678725 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:37.678728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:37.678775 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:37.678850 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:37.679328 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.680122 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.681344 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.681894 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.682038 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.682101 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:37.682789 (+ 688us) server_negotiation.cc:299] Negotiation successful
0808 04:34:37.682872 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 04:34:37.683488 18092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70df860e03ed4caab4b7dfa4aa735e57" candidate_uuid: "0528dba3541b426cacdbf67126d9851d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e705f331044f1faf9f93230e88ac35" is_pre_election: true
I20260808 04:34:37.683669 18092 raft_consensus.cc:2454] T 70df860e03ed4caab4b7dfa4aa735e57 P 57e705f331044f1faf9f93230e88ac35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0528dba3541b426cacdbf67126d9851d in term 0.
I20260808 04:34:37.683835 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.678106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:36353 (local address 127.15.228.66:39509)
0808 04:34:37.678281 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:37.678287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:37.679152 (+ 865us) server_negotiation.cc:407] Connection header received
0808 04:34:37.679367 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:37.679372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:37.679427 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:37.679480 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:37.679855 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.680413 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.682363 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.682905 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.683067 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.683134 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:37.683647 (+ 513us) server_negotiation.cc:299] Negotiation successful
0808 04:34:37.683718 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 04:34:37.683843 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.677820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:39509 (local address 127.15.228.67:36353)
0808 04:34:37.679052 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:37.679068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:37.679229 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:37.679619 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:37.679621 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:37.679636 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:37.679733 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.679737 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.680530 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.680534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.682230 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.682242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.683392 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.683395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.683512 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.683531 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 04:34:37.683629 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 04:34:37.683700 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1128,"mutex_wait_us":54,"thread_start_us":138,"threads_started":1}
I20260808 04:34:37.684360 18415 leader_election.cc:302] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0528dba3541b426cacdbf67126d9851d, 57e705f331044f1faf9f93230e88ac35; no voters:
I20260808 04:34:37.684530 18307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70df860e03ed4caab4b7dfa4aa735e57" candidate_uuid: "0528dba3541b426cacdbf67126d9851d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a92ec8c7f354da8937e305553b4553c" is_pre_election: true
I20260808 04:34:37.684778 18307 raft_consensus.cc:2454] T 70df860e03ed4caab4b7dfa4aa735e57 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0528dba3541b426cacdbf67126d9851d in term 0.
I20260808 04:34:37.684842 18626 raft_consensus.cc:2781] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:34:37.684901 18626 raft_consensus.cc:482] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:34:37.684940 18626 raft_consensus.cc:3037] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:37.686152 18626 raft_consensus.cc:504] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.686573 18626 leader_election.cc:288] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [CANDIDATE]: Term 1 election: Requested vote from peers 57e705f331044f1faf9f93230e88ac35 (127.15.228.65:45189), 2a92ec8c7f354da8937e305553b4553c (127.15.228.66:39509)
I20260808 04:34:37.687075 18092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70df860e03ed4caab4b7dfa4aa735e57" candidate_uuid: "0528dba3541b426cacdbf67126d9851d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e705f331044f1faf9f93230e88ac35"
I20260808 04:34:37.687099 18307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70df860e03ed4caab4b7dfa4aa735e57" candidate_uuid: "0528dba3541b426cacdbf67126d9851d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a92ec8c7f354da8937e305553b4553c"
I20260808 04:34:37.687201 18092 raft_consensus.cc:3037] T 70df860e03ed4caab4b7dfa4aa735e57 P 57e705f331044f1faf9f93230e88ac35 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:37.687227 18307 raft_consensus.cc:3037] T 70df860e03ed4caab4b7dfa4aa735e57 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:37.688432 18307 raft_consensus.cc:2454] T 70df860e03ed4caab4b7dfa4aa735e57 P 2a92ec8c7f354da8937e305553b4553c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0528dba3541b426cacdbf67126d9851d in term 1.
I20260808 04:34:37.688567 18092 raft_consensus.cc:2454] T 70df860e03ed4caab4b7dfa4aa735e57 P 57e705f331044f1faf9f93230e88ac35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0528dba3541b426cacdbf67126d9851d in term 1.
I20260808 04:34:37.688786 18414 leader_election.cc:302] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0528dba3541b426cacdbf67126d9851d, 2a92ec8c7f354da8937e305553b4553c; no voters:
I20260808 04:34:37.689030 18626 raft_consensus.cc:2781] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:34:37.689275 18626 raft_consensus.cc:686] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [term 1 LEADER]: Becoming Leader. State: Replica: 0528dba3541b426cacdbf67126d9851d, State: Running, Role: LEADER
I20260808 04:34:37.689554 18626 consensus_queue.cc:237] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.692636 17863 catalog_manager.cc:5495] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0528dba3541b426cacdbf67126d9851d (127.15.228.67). New cstate: current_term: 1 leader_uuid: "0528dba3541b426cacdbf67126d9851d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } health_report { overall_health: HEALTHY } } }
Aug 08 04:34:37 dist-test-slave-drl0 krb5kdc[17805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163676, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:37.735636 18672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.725253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:43123 (local address 127.0.0.1:51162)
0808 04:34:37.725778 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:37.725794 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:37.725972 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:37.726243 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:37.726246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:37.726728 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:37.726826 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.726832 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.727611 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.727616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.728162 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:37.728169 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.728600 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.728603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:37.728846 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.729944 (+ 1098us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:37.729956 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:37.731604 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:37.733722 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:37.733735 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:37.733743 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:37.734091 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:37.734417 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:37.734420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:37.734422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:37.734508 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:37.734919 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:37.734943 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:37.735065 (+ 122us) client_negotiation.cc:769] Sending connection context
0808 04:34:37.735192 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 04:34:37.735418 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":234,"threads_started":1}
I20260808 04:34:37.735636 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:37.725193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51162 (local address 127.15.228.126:43123)
0808 04:34:37.725347 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:37.725351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:37.725905 (+ 554us) server_negotiation.cc:407] Connection header received
0808 04:34:37.726082 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:37.726085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:37.726129 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:37.726215 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:37.726980 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.727501 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.728278 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:37.728487 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:37.728597 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:37.731767 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:37.731771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:37.731773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:37.731794 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:37.733553 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:37.734192 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:37.734197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:37.734199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:37.734306 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:37.734607 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:37.734610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:37.734612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:37.734803 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:37.734919 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:37.735240 (+ 321us) server_negotiation.cc:299] Negotiation successful
0808 04:34:37.735427 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":56}
I20260808 04:34:37.741583 17863 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:51162:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 04:34:37.742682 17863 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:34:37.749193 18276 tablet_service.cc:1467] Processing CreateTablet for tablet 1da0054c5a2c430eb6536170dace1433 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9c5905e9f49c4e789373af1bb2ea7aee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:34:37.749298 18499 tablet_service.cc:1467] Processing CreateTablet for tablet 1da0054c5a2c430eb6536170dace1433 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9c5905e9f49c4e789373af1bb2ea7aee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:34:37.749128 18062 tablet_service.cc:1467] Processing CreateTablet for tablet 1da0054c5a2c430eb6536170dace1433 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9c5905e9f49c4e789373af1bb2ea7aee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:34:37.749648 18499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1da0054c5a2c430eb6536170dace1433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:37.749648 18276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1da0054c5a2c430eb6536170dace1433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:37.749891 18062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1da0054c5a2c430eb6536170dace1433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:37.752784 18618 tablet_bootstrap.cc:492] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c: Bootstrap starting.
I20260808 04:34:37.754043 18618 tablet_bootstrap.cc:654] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:37.756110 18621 tablet_bootstrap.cc:492] T 1da0054c5a2c430eb6536170dace1433 P 0528dba3541b426cacdbf67126d9851d: Bootstrap starting.
I20260808 04:34:37.756176 18618 tablet_bootstrap.cc:492] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c: No bootstrap required, opened a new log
I20260808 04:34:37.756263 18618 ts_tablet_manager.cc:1397] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260808 04:34:37.756866 18618 raft_consensus.cc:348] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.756987 18618 raft_consensus.cc:374] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:37.757018 18618 raft_consensus.cc:729] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a92ec8c7f354da8937e305553b4553c, State: Initialized, Role: FOLLOWER
I20260808 04:34:37.757171 18618 consensus_queue.cc:260] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.757416 18630 tablet_replica.cc:439] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } } }
I20260808 04:34:37.757545 18630 tablet_replica.cc:442] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:34:37.757712 18621 tablet_bootstrap.cc:654] T 1da0054c5a2c430eb6536170dace1433 P 0528dba3541b426cacdbf67126d9851d: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:37.757761 18618 ts_tablet_manager.cc:1428] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:34:37.759552 18622 tablet_bootstrap.cc:492] T 1da0054c5a2c430eb6536170dace1433 P 57e705f331044f1faf9f93230e88ac35: Bootstrap starting.
I20260808 04:34:37.760076 18621 tablet_bootstrap.cc:492] T 1da0054c5a2c430eb6536170dace1433 P 0528dba3541b426cacdbf67126d9851d: No bootstrap required, opened a new log
I20260808 04:34:37.760174 18621 ts_tablet_manager.cc:1397] T 1da0054c5a2c430eb6536170dace1433 P 0528dba3541b426cacdbf67126d9851d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260808 04:34:37.760742 18622 tablet_bootstrap.cc:654] T 1da0054c5a2c430eb6536170dace1433 P 57e705f331044f1faf9f93230e88ac35: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:37.760720 18621 raft_consensus.cc:348] T 1da0054c5a2c430eb6536170dace1433 P 0528dba3541b426cacdbf67126d9851d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.760869 18621 raft_consensus.cc:374] T 1da0054c5a2c430eb6536170dace1433 P 0528dba3541b426cacdbf67126d9851d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:37.760910 18621 raft_consensus.cc:729] T 1da0054c5a2c430eb6536170dace1433 P 0528dba3541b426cacdbf67126d9851d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0528dba3541b426cacdbf67126d9851d, State: Initialized, Role: FOLLOWER
I20260808 04:34:37.761072 18621 consensus_queue.cc:260] T 1da0054c5a2c430eb6536170dace1433 P 0528dba3541b426cacdbf67126d9851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.761472 18626 tablet_replica.cc:439] T 1da0054c5a2c430eb6536170dace1433 P 0528dba3541b426cacdbf67126d9851d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } } }
I20260808 04:34:37.761600 18626 tablet_replica.cc:442] T 1da0054c5a2c430eb6536170dace1433 P 0528dba3541b426cacdbf67126d9851d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:34:37.763062 18622 tablet_bootstrap.cc:492] T 1da0054c5a2c430eb6536170dace1433 P 57e705f331044f1faf9f93230e88ac35: No bootstrap required, opened a new log
I20260808 04:34:37.763160 18622 ts_tablet_manager.cc:1397] T 1da0054c5a2c430eb6536170dace1433 P 57e705f331044f1faf9f93230e88ac35: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260808 04:34:37.763645 18621 ts_tablet_manager.cc:1428] T 1da0054c5a2c430eb6536170dace1433 P 0528dba3541b426cacdbf67126d9851d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:34:37.763737 18622 raft_consensus.cc:348] T 1da0054c5a2c430eb6536170dace1433 P 57e705f331044f1faf9f93230e88ac35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.763862 18622 raft_consensus.cc:374] T 1da0054c5a2c430eb6536170dace1433 P 57e705f331044f1faf9f93230e88ac35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:37.763903 18622 raft_consensus.cc:729] T 1da0054c5a2c430eb6536170dace1433 P 57e705f331044f1faf9f93230e88ac35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e705f331044f1faf9f93230e88ac35, State: Initialized, Role: FOLLOWER
I20260808 04:34:37.764138 18622 consensus_queue.cc:260] T 1da0054c5a2c430eb6536170dace1433 P 57e705f331044f1faf9f93230e88ac35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:37.764425 18628 tablet_replica.cc:439] T 1da0054c5a2c430eb6536170dace1433 P 57e705f331044f1faf9f93230e88ac35: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } } }
I20260808 04:34:37.764559 18628 tablet_replica.cc:442] T 1da0054c5a2c430eb6536170dace1433 P 57e705f331044f1faf9f93230e88ac35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:34:37.764571 18622 ts_tablet_manager.cc:1428] T 1da0054c5a2c430eb6536170dace1433 P 57e705f331044f1faf9f93230e88ac35: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:34:38.030475 18630 raft_consensus.cc:482] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:34:38.030643 18630 raft_consensus.cc:504] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:38.031181 18630 leader_election.cc:288] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57e705f331044f1faf9f93230e88ac35 (127.15.228.65:45189), 0528dba3541b426cacdbf67126d9851d (127.15.228.67:39899)
I20260808 04:34:38.031762 18092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1da0054c5a2c430eb6536170dace1433" candidate_uuid: "2a92ec8c7f354da8937e305553b4553c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e705f331044f1faf9f93230e88ac35" is_pre_election: true
I20260808 04:34:38.031965 18092 raft_consensus.cc:2454] T 1da0054c5a2c430eb6536170dace1433 P 57e705f331044f1faf9f93230e88ac35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a92ec8c7f354da8937e305553b4553c in term 0.
I20260808 04:34:38.031994 18528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1da0054c5a2c430eb6536170dace1433" candidate_uuid: "2a92ec8c7f354da8937e305553b4553c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0528dba3541b426cacdbf67126d9851d" is_pre_election: true
I20260808 04:34:38.032166 18528 raft_consensus.cc:2454] T 1da0054c5a2c430eb6536170dace1433 P 0528dba3541b426cacdbf67126d9851d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a92ec8c7f354da8937e305553b4553c in term 0.
I20260808 04:34:38.032348 18201 leader_election.cc:302] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a92ec8c7f354da8937e305553b4553c, 57e705f331044f1faf9f93230e88ac35; no voters:
I20260808 04:34:38.032564 18630 raft_consensus.cc:2781] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:34:38.032626 18630 raft_consensus.cc:482] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:34:38.032658 18630 raft_consensus.cc:3037] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:38.033949 18630 raft_consensus.cc:504] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:38.034523 18630 leader_election.cc:288] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c [CANDIDATE]: Term 1 election: Requested vote from peers 57e705f331044f1faf9f93230e88ac35 (127.15.228.65:45189), 0528dba3541b426cacdbf67126d9851d (127.15.228.67:39899)
I20260808 04:34:38.034802 18092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1da0054c5a2c430eb6536170dace1433" candidate_uuid: "2a92ec8c7f354da8937e305553b4553c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e705f331044f1faf9f93230e88ac35"
I20260808 04:34:38.034912 18092 raft_consensus.cc:3037] T 1da0054c5a2c430eb6536170dace1433 P 57e705f331044f1faf9f93230e88ac35 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:38.035673 18528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1da0054c5a2c430eb6536170dace1433" candidate_uuid: "2a92ec8c7f354da8937e305553b4553c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0528dba3541b426cacdbf67126d9851d"
I20260808 04:34:38.035903 18528 raft_consensus.cc:3037] T 1da0054c5a2c430eb6536170dace1433 P 0528dba3541b426cacdbf67126d9851d [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:38.036722 18092 raft_consensus.cc:2454] T 1da0054c5a2c430eb6536170dace1433 P 57e705f331044f1faf9f93230e88ac35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a92ec8c7f354da8937e305553b4553c in term 1.
I20260808 04:34:38.037122 18201 leader_election.cc:302] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a92ec8c7f354da8937e305553b4553c, 57e705f331044f1faf9f93230e88ac35; no voters:
I20260808 04:34:38.037318 18630 raft_consensus.cc:2781] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:34:38.037413 18630 raft_consensus.cc:686] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c [term 1 LEADER]: Becoming Leader. State: Replica: 2a92ec8c7f354da8937e305553b4553c, State: Running, Role: LEADER
I20260808 04:34:38.037603 18528 raft_consensus.cc:2454] T 1da0054c5a2c430eb6536170dace1433 P 0528dba3541b426cacdbf67126d9851d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a92ec8c7f354da8937e305553b4553c in term 1.
I20260808 04:34:38.037586 18630 consensus_queue.cc:237] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } }
I20260808 04:34:38.038295 18627 tablet_replica.cc:439] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c: TxnStatusTablet state changed. Reason: New leader 2a92ec8c7f354da8937e305553b4553c. Latest consensus state: current_term: 1 leader_uuid: "2a92ec8c7f354da8937e305553b4553c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } } }
I20260808 04:34:38.038408 18627 tablet_replica.cc:442] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:34:38.038704 18677 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 04:34:38.038969 18677 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 04:34:38.039499 17863 catalog_manager.cc:5495] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a92ec8c7f354da8937e305553b4553c (127.15.228.66). New cstate: current_term: 1 leader_uuid: "2a92ec8c7f354da8937e305553b4553c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } health_report { overall_health: UNKNOWN } } }
I20260808 04:34:38.064805 18672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:38.060769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:39509 (local address 127.0.0.1:43304)
0808 04:34:38.061016 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:38.061032 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:38.061133 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:38.061477 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:38.061480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:38.061487 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:34:38.061590 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:38.061594 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:38.062650 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:38.062653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:38.063420 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:38.063430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:38.064007 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:38.064010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:38.064081 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:38.064135 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:34:38.064484 (+ 349us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:34:38.064502 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 04:34:38.064599 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 04:34:38.064677 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260808 04:34:38.064826 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:38.060901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.15.228.66:39509)
0808 04:34:38.061065 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:38.061070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:38.061086 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:34:38.061256 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:38.061259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:38.061315 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:38.061408 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:34:38.061698 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:38.062500 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:38.063539 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:38.063882 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:38.063987 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:38.064270 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:34:38.064363 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:34:38.064440 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:38.064629 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 04:34:38.064684 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 04:34:38.068184 18528 raft_consensus.cc:1261] T 1da0054c5a2c430eb6536170dace1433 P 0528dba3541b426cacdbf67126d9851d [term 1 FOLLOWER]: Refusing update from remote peer 2a92ec8c7f354da8937e305553b4553c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:34:38.068336 18092 raft_consensus.cc:1261] T 1da0054c5a2c430eb6536170dace1433 P 57e705f331044f1faf9f93230e88ac35 [term 1 FOLLOWER]: Refusing update from remote peer 2a92ec8c7f354da8937e305553b4553c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:34:38.068908 18627 consensus_queue.cc:1035] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c [LEADER]: Connected to new peer: Peer: permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:34:38.069123 18630 consensus_queue.cc:1035] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:34:38.071084 18628 tablet_replica.cc:439] T 1da0054c5a2c430eb6536170dace1433 P 57e705f331044f1faf9f93230e88ac35: TxnStatusTablet state changed. Reason: New leader 2a92ec8c7f354da8937e305553b4553c. Latest consensus state: current_term: 1 leader_uuid: "2a92ec8c7f354da8937e305553b4553c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } } }
I20260808 04:34:38.071216 18628 tablet_replica.cc:442] T 1da0054c5a2c430eb6536170dace1433 P 57e705f331044f1faf9f93230e88ac35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:34:38.073189 18630 tablet_replica.cc:439] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a92ec8c7f354da8937e305553b4553c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } } }
I20260808 04:34:38.073297 18630 tablet_replica.cc:442] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:34:38.073388 18634 mvcc.cc:205] Tried to move back new op lower bound from 7316126425360977920 to 7316126425243480064. Current Snapshot: MvccSnapshot[applied={T|T < 7316126425360977920}]
I20260808 04:34:38.074456 18628 tablet_replica.cc:439] T 1da0054c5a2c430eb6536170dace1433 P 57e705f331044f1faf9f93230e88ac35: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a92ec8c7f354da8937e305553b4553c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } } }
I20260808 04:34:38.074520 18635 mvcc.cc:205] Tried to move back new op lower bound from 7316126425360977920 to 7316126425243480064. Current Snapshot: MvccSnapshot[applied={T|T < 7316126425360977920}]
I20260808 04:34:38.074572 18628 tablet_replica.cc:442] T 1da0054c5a2c430eb6536170dace1433 P 57e705f331044f1faf9f93230e88ac35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:34:38.074575 18626 tablet_replica.cc:439] T 1da0054c5a2c430eb6536170dace1433 P 0528dba3541b426cacdbf67126d9851d: TxnStatusTablet state changed. Reason: New leader 2a92ec8c7f354da8937e305553b4553c. Latest consensus state: current_term: 1 leader_uuid: "2a92ec8c7f354da8937e305553b4553c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } } }
I20260808 04:34:38.074713 18626 tablet_replica.cc:442] T 1da0054c5a2c430eb6536170dace1433 P 0528dba3541b426cacdbf67126d9851d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:34:38.078696 18627 tablet_replica.cc:439] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a92ec8c7f354da8937e305553b4553c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } } }
I20260808 04:34:38.078802 18627 tablet_replica.cc:442] T 1da0054c5a2c430eb6536170dace1433 P 2a92ec8c7f354da8937e305553b4553c: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:34:38.079944 18638 mvcc.cc:205] Tried to move back new op lower bound from 7316126425360977920 to 7316126425243480064. Current Snapshot: MvccSnapshot[applied={T|T < 7316126425360977920}]
I20260808 04:34:38.080173 18626 tablet_replica.cc:439] T 1da0054c5a2c430eb6536170dace1433 P 0528dba3541b426cacdbf67126d9851d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a92ec8c7f354da8937e305553b4553c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 } } peers { permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 } } peers { permanent_uuid: "0528dba3541b426cacdbf67126d9851d" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39899 } } }
I20260808 04:34:38.080267 18626 tablet_replica.cc:442] T 1da0054c5a2c430eb6536170dace1433 P 0528dba3541b426cacdbf67126d9851d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:34:38.087103 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:38.080474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.15.228.67:39899)
0808 04:34:38.083158 (+ 2684us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:38.083162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:38.083177 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:34:38.083244 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:38.083246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:38.083305 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:38.083467 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:34:38.083760 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:38.084414 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:38.085348 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:38.085612 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:38.085742 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:38.086060 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:34:38.086541 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:34:38.086686 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:38.086771 (+ 85us) server_negotiation.cc:299] Negotiation successful
0808 04:34:38.086831 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2613,"thread_start_us":84,"threads_started":1}
I20260808 04:34:38.100157 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:38.095435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:49787 (local address 127.15.228.66:39509)
0808 04:34:38.095562 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:38.095566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:38.095787 (+ 221us) server_negotiation.cc:407] Connection header received
0808 04:34:38.095902 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:38.095904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:38.095951 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:38.096019 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:34:38.096479 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:38.096984 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:38.098447 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:38.098670 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:38.098893 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:38.099634 (+ 741us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:34:38.099717 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:34:38.099798 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:38.099964 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 04:34:38.100031 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:34:38.100274 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:38.095250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:39509 (local address 127.15.228.67:49787)
0808 04:34:38.095640 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:38.095657 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:38.095795 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:38.096056 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:38.096060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:38.096068 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:34:38.096318 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:38.096326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:38.097097 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:38.097101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:38.098265 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:38.098277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:38.099057 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:38.099064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:38.099288 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:38.099386 (+ 98us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:34:38.099832 (+ 446us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:34:38.099840 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 04:34:38.099921 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 04:34:38.100045 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":135,"threads_started":1}
I20260808 04:34:38.108826 18092 raft_consensus.cc:1261] T 70df860e03ed4caab4b7dfa4aa735e57 P 57e705f331044f1faf9f93230e88ac35 [term 1 FOLLOWER]: Refusing update from remote peer 0528dba3541b426cacdbf67126d9851d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:34:38.109182 18307 raft_consensus.cc:1261] T 70df860e03ed4caab4b7dfa4aa735e57 P 2a92ec8c7f354da8937e305553b4553c [term 1 FOLLOWER]: Refusing update from remote peer 0528dba3541b426cacdbf67126d9851d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:34:38.109546 18663 consensus_queue.cc:1035] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "57e705f331044f1faf9f93230e88ac35" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:34:38.109800 18626 consensus_queue.cc:1035] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a92ec8c7f354da8937e305553b4553c" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:34:38.140331 18694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:38.135454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:39899 (local address 127.15.228.66:54825)
0808 04:34:38.136022 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:38.136040 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:38.136213 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:38.136762 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:38.136766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:38.136777 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:34:38.136919 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:38.136925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:38.137834 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:38.137838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:38.138779 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:38.138790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:38.139411 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:38.139415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:38.139511 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:38.139574 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:34:38.140038 (+ 464us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:34:38.140047 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 04:34:38.140111 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 04:34:38.140202 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":262,"threads_started":1}
I20260808 04:34:38.140329 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:38.135608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:54825 (local address 127.15.228.67:39899)
0808 04:34:38.135749 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:38.135753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:38.136124 (+ 371us) server_negotiation.cc:407] Connection header received
0808 04:34:38.136389 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:38.136393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:38.136475 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:38.136599 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:34:38.137123 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:38.137667 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:38.139009 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:38.139279 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:38.139416 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:38.139840 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:34:38.139938 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:34:38.140028 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:38.140121 (+ 93us) server_negotiation.cc:299] Negotiation successful
0808 04:34:38.140175 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:34:38.173384 17863 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:34:38.173571 17863 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:34:38.176777 17863 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d74e5ff2e7a3460a9fcbab63bfd9fdcd: Sending DeleteTablet for 3 replicas of tablet 70df860e03ed4caab4b7dfa4aa735e57
I20260808 04:34:38.177570 18062 tablet_service.cc:1514] Processing DeleteTablet for tablet 70df860e03ed4caab4b7dfa4aa735e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:38 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:37058
I20260808 04:34:38.178025 18499 tablet_service.cc:1514] Processing DeleteTablet for tablet 70df860e03ed4caab4b7dfa4aa735e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:38 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:48700
I20260808 04:34:38.178081 18698 tablet_replica.cc:330] T 70df860e03ed4caab4b7dfa4aa735e57 P 57e705f331044f1faf9f93230e88ac35: stopping tablet replica
I20260808 04:34:38.178400 18698 raft_consensus.cc:2229] T 70df860e03ed4caab4b7dfa4aa735e57 P 57e705f331044f1faf9f93230e88ac35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:34:38.178613 18698 raft_consensus.cc:2258] T 70df860e03ed4caab4b7dfa4aa735e57 P 57e705f331044f1faf9f93230e88ac35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:34:38.179509 18698 ts_tablet_manager.cc:1905] T 70df860e03ed4caab4b7dfa4aa735e57 P 57e705f331044f1faf9f93230e88ac35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:34:38.179661 18276 tablet_service.cc:1514] Processing DeleteTablet for tablet 70df860e03ed4caab4b7dfa4aa735e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:38 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:43286
I20260808 04:34:38.180830 18699 tablet_replica.cc:330] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d: stopping tablet replica
I20260808 04:34:38.180980 18699 raft_consensus.cc:2229] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:34:38.181295 18699 raft_consensus.cc:2258] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:34:38.181981 18699 ts_tablet_manager.cc:1905] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:34:38.183699 18698 ts_tablet_manager.cc:1918] T 70df860e03ed4caab4b7dfa4aa735e57 P 57e705f331044f1faf9f93230e88ac35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:34:38.183833 18698 log.cc:1196] T 70df860e03ed4caab4b7dfa4aa735e57 P 57e705f331044f1faf9f93230e88ac35: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-0/wal/wals/70df860e03ed4caab4b7dfa4aa735e57
I20260808 04:34:38.184206 18698 ts_tablet_manager.cc:1939] T 70df860e03ed4caab4b7dfa4aa735e57 P 57e705f331044f1faf9f93230e88ac35: Deleting consensus metadata
I20260808 04:34:38.185935 17849 catalog_manager.cc:4841] TS 57e705f331044f1faf9f93230e88ac35 (127.15.228.65:45189): tablet 70df860e03ed4caab4b7dfa4aa735e57 (table test-table [id=9f44b0f5741b4eeb9912bb2c7f0d1a1d]) successfully deleted
I20260808 04:34:38.186269 18699 ts_tablet_manager.cc:1918] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:34:38.186362 18699 log.cc:1196] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/70df860e03ed4caab4b7dfa4aa735e57
I20260808 04:34:38.186300 18701 tablet_replica.cc:330] T 70df860e03ed4caab4b7dfa4aa735e57 P 2a92ec8c7f354da8937e305553b4553c: stopping tablet replica
I20260808 04:34:38.186573 18701 raft_consensus.cc:2229] T 70df860e03ed4caab4b7dfa4aa735e57 P 2a92ec8c7f354da8937e305553b4553c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:34:38.186651 18699 ts_tablet_manager.cc:1939] T 70df860e03ed4caab4b7dfa4aa735e57 P 0528dba3541b426cacdbf67126d9851d: Deleting consensus metadata
I20260808 04:34:38.186841 18701 raft_consensus.cc:2258] T 70df860e03ed4caab4b7dfa4aa735e57 P 2a92ec8c7f354da8937e305553b4553c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:34:38.187458 17850 catalog_manager.cc:4841] TS 0528dba3541b426cacdbf67126d9851d (127.15.228.67:39899): tablet 70df860e03ed4caab4b7dfa4aa735e57 (table test-table [id=9f44b0f5741b4eeb9912bb2c7f0d1a1d]) successfully deleted
I20260808 04:34:38.188431 18701 ts_tablet_manager.cc:1905] T 70df860e03ed4caab4b7dfa4aa735e57 P 2a92ec8c7f354da8937e305553b4553c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:34:38.193531 18701 ts_tablet_manager.cc:1918] T 70df860e03ed4caab4b7dfa4aa735e57 P 2a92ec8c7f354da8937e305553b4553c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:34:38.193701 18701 log.cc:1196] T 70df860e03ed4caab4b7dfa4aa735e57 P 2a92ec8c7f354da8937e305553b4553c: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/70df860e03ed4caab4b7dfa4aa735e57
I20260808 04:34:38.194062 18701 ts_tablet_manager.cc:1939] T 70df860e03ed4caab4b7dfa4aa735e57 P 2a92ec8c7f354da8937e305553b4553c: Deleting consensus metadata
I20260808 04:34:38.194900 17846 catalog_manager.cc:4841] TS 2a92ec8c7f354da8937e305553b4553c (127.15.228.66:39509): tablet 70df860e03ed4caab4b7dfa4aa735e57 (table test-table [id=9f44b0f5741b4eeb9912bb2c7f0d1a1d]) successfully deleted
Aug 08 04:34:38 dist-test-slave-drl0 krb5kdc[17805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163677, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.228.65@KRBTEST.COM
I20260808 04:34:38.201110 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:38.180225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37062 (local address 127.15.228.65:45189)
0808 04:34:38.180341 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:38.180346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:38.187914 (+ 7568us) server_negotiation.cc:407] Connection header received
0808 04:34:38.188110 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:38.188113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:38.188164 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:38.188227 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:38.189422 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:38.190206 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:38.191517 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:38.191853 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:38.191938 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:38.196279 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:38.196295 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:38.196315 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:38.196346 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:38.198582 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:38.199185 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:38.199192 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:38.199194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:38.199327 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:38.199674 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:38.199678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:38.199679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:38.200031 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:38.200263 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:38.200685 (+ 422us) server_negotiation.cc:299] Negotiation successful
0808 04:34:38.200829 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260808 04:34:38.202262 18151 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37062
I20260808 04:34:38.216184 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:38.204419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51176 (local address 127.15.228.126:43123)
0808 04:34:38.204591 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:38.204596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:38.204945 (+ 349us) server_negotiation.cc:407] Connection header received
0808 04:34:38.205041 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:38.205044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:38.205100 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:38.205193 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:38.206046 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:38.206883 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:38.208626 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:38.209047 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:38.209204 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:38.211327 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:38.211332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:38.211335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:38.211372 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:38.213940 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:38.214522 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:38.214526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:38.214528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:38.214650 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:38.215055 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:38.215059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:38.215061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:38.215313 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:38.215479 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:38.215868 (+ 389us) server_negotiation.cc:299] Negotiation successful
0808 04:34:38.216011 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260808 04:34:38.217190 17863 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51176
I20260808 04:34:38.218731 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 17973
I20260808 04:34:38.236430 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 18173
I20260808 04:34:38.250052 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 18385
I20260808 04:34:38.263124 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 17823
2026-08-08T04:34:38Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3427 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:34:38 dist-test-slave-drl0 krb5kdc[18710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:34:38 dist-test-slave-drl0 krb5kdc[18710](info): set up 2 sockets
Aug 08 04:34:38 dist-test-slave-drl0 krb5kdc[18710](info): commencing operation
krb5kdc: starting...
W20260808 04:34:39.568029 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.278s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:34:39 dist-test-slave-drl0 krb5kdc[18710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:34:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:34:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:34:39.733016 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:36681
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:36681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:39.899356 18725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:39.899665 18725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:39.899711 18725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:39.904592 18725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:34:39.904659 18725 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 04:34:39.904678 18725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:39.904697 18725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:39.904713 18725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:34:39.904729 18725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:34:39.915153 18725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:36681
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:36681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:39.916879 18725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:39.918457 18725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:39.930434 18733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:39.930521 18735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:39.930677 18725 server_base.cc:1034] running on GCE node
W20260808 04:34:39.930522 18732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:39.931808 18725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:39.933795 18725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:39.935060 18725 hybrid_clock.cc:648] HybridClock initialized: now 1786163679935005 us; error 71 us; skew 500 ppm
Aug 08 04:34:39 dist-test-slave-drl0 krb5kdc[18710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163679, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:39.938388 18725 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:34:39.939559 18725 webserver.cc:460] Webserver started at http://127.15.228.126:36751/ using document root <none> and password file <none>
I20260808 04:34:39.940182 18725 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:39.940248 18725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:39.940487 18725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:39.942780 18725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "111dbf0a4f274bb390450a906a6879c3"
format_stamp: "Formatted at 2026-08-08 04:34:39 on dist-test-slave-drl0"
server_key: "94d048e603bc4f69fc80cb8bec99b44a"
server_key_iv: "a07064fb1b9335f266f9246e24acc0d1"
server_key_version: "encryptionkey@0"
I20260808 04:34:39.943470 18725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "111dbf0a4f274bb390450a906a6879c3"
format_stamp: "Formatted at 2026-08-08 04:34:39 on dist-test-slave-drl0"
server_key: "94d048e603bc4f69fc80cb8bec99b44a"
server_key_iv: "a07064fb1b9335f266f9246e24acc0d1"
server_key_version: "encryptionkey@0"
I20260808 04:34:39.947769 18725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:34:39.950896 18744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:39.952761 18725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:34:39.952898 18725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "111dbf0a4f274bb390450a906a6879c3"
format_stamp: "Formatted at 2026-08-08 04:34:39 on dist-test-slave-drl0"
server_key: "94d048e603bc4f69fc80cb8bec99b44a"
server_key_iv: "a07064fb1b9335f266f9246e24acc0d1"
server_key_version: "encryptionkey@0"
I20260808 04:34:39.953080 18725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:39.966853 18725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:39.970448 18725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:39.970652 18725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:39.979910 18725 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:36681
I20260808 04:34:39.979981 18830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:36681 every 8 connection(s)
I20260808 04:34:39.980986 18725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:34:39.984071 18832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:39.986251 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 18725
I20260808 04:34:39.986380 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:34:39.986675 16273 external_mini_cluster.cc:1428] Setting key befa62cc29966543d6aae1a1c6b39e60
I20260808 04:34:39.990968 18832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3: Bootstrap starting.
I20260808 04:34:39.993600 18832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:39.994661 18832 log.cc:824] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:39.997059 18832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3: No bootstrap required, opened a new log
Aug 08 04:34:39 dist-test-slave-drl0 krb5kdc[18710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:40.001147 18832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "111dbf0a4f274bb390450a906a6879c3" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36681 } }
I20260808 04:34:40.001435 18832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:40.001539 18832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111dbf0a4f274bb390450a906a6879c3, State: Initialized, Role: FOLLOWER
I20260808 04:34:40.002108 18832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "111dbf0a4f274bb390450a906a6879c3" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36681 } }
I20260808 04:34:40.002319 18832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:34:40.002417 18832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:34:40.002521 18832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:40.003587 18832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "111dbf0a4f274bb390450a906a6879c3" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36681 } }
I20260808 04:34:40.003958 18832 leader_election.cc:302] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 111dbf0a4f274bb390450a906a6879c3; no voters:
I20260808 04:34:40.004364 18832 leader_election.cc:288] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:34:40.004426 18838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:34:40.004653 18838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3 [term 1 LEADER]: Becoming Leader. State: Replica: 111dbf0a4f274bb390450a906a6879c3, State: Running, Role: LEADER
I20260808 04:34:40.004984 18838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "111dbf0a4f274bb390450a906a6879c3" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36681 } }
I20260808 04:34:40.005697 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:39.988627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60586 (local address 127.15.228.126:36681)
0808 04:34:39.989077 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:39.989093 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:39.989129 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:34:39.989823 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:39.989839 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:39.990171 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:39.990509 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:39.991656 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:39.993102 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:39.994590 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:39.995420 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:39.995958 (+ 538us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:39.999174 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:39.999187 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:39.999214 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:39.999274 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:40.001383 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:40.002210 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:40.002216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:40.002221 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:40.002520 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:40.003005 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:40.003008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:40.003010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:40.003458 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:40.003715 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:40.003970 (+ 255us) server_negotiation.cc:299] Negotiation successful
0808 04:34:40.004324 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":152,"threads_started":1}
I20260808 04:34:40.005766 18832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:34:40.006712 18838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "111dbf0a4f274bb390450a906a6879c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "111dbf0a4f274bb390450a906a6879c3" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36681 } } }
I20260808 04:34:40.006884 18838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3 [sys.catalog]: This master's current role is: LEADER
I20260808 04:34:40.008082 18844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:34:40.010509 18840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 111dbf0a4f274bb390450a906a6879c3. Latest consensus state: current_term: 1 leader_uuid: "111dbf0a4f274bb390450a906a6879c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "111dbf0a4f274bb390450a906a6879c3" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36681 } } }
I20260808 04:34:40.010654 18840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3 [sys.catalog]: This master's current role is: LEADER
I20260808 04:34:40.013059 18844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:34:40.020689 18844 catalog_manager.cc:1347] Generated new cluster ID: ad965d592428448db3d40efb582bab62
I20260808 04:34:40.020784 18844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:34:40.035089 18844 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:34:40.036273 18844 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:34:40.042208 18844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3: Generated new TSK 0
I20260808 04:34:40.043072 18844 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
I20260808 04:34:40.091369 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:36681
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:40.252599 18870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:40.252911 18870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:40.252969 18870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:40.257683 18870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:40.257750 18870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:40.257792 18870 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 04:34:40.257807 18870 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:34:40.257893 18870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:34:40.264050 18870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.228.126:36681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:40.265246 18870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:40.266379 18870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:40.279480 18879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:40.279486 18876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:40.279486 18877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:40.280192 18870 server_base.cc:1034] running on GCE node
I20260808 04:34:40.280674 18870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:40.281432 18870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:40.282653 18870 hybrid_clock.cc:648] HybridClock initialized: now 1786163680282621 us; error 46 us; skew 500 ppm
Aug 08 04:34:40 dist-test-slave-drl0 krb5kdc[18710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163680, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:40.286700 18870 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:34:40.288942 18870 webserver.cc:460] Webserver started at http://127.15.228.65:38685/ using document root <none> and password file <none>
I20260808 04:34:40.289847 18870 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:40.289965 18870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:40.290288 18870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:40.294989 18870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "bf60ee870f894a13beef385cf9a3d420"
format_stamp: "Formatted at 2026-08-08 04:34:40 on dist-test-slave-drl0"
server_key: "90e3674894d11f1b7a15f216eaeea33b"
server_key_iv: "4dea3916daa2c78ff8f34c1d3a526879"
server_key_version: "encryptionkey@0"
I20260808 04:34:40.295883 18870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "bf60ee870f894a13beef385cf9a3d420"
format_stamp: "Formatted at 2026-08-08 04:34:40 on dist-test-slave-drl0"
server_key: "90e3674894d11f1b7a15f216eaeea33b"
server_key_iv: "4dea3916daa2c78ff8f34c1d3a526879"
server_key_version: "encryptionkey@0"
I20260808 04:34:40.301625 18870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:34:40.304957 18887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:40.306733 18870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:34:40.306912 18870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "bf60ee870f894a13beef385cf9a3d420"
format_stamp: "Formatted at 2026-08-08 04:34:40 on dist-test-slave-drl0"
server_key: "90e3674894d11f1b7a15f216eaeea33b"
server_key_iv: "4dea3916daa2c78ff8f34c1d3a526879"
server_key_version: "encryptionkey@0"
I20260808 04:34:40.307147 18870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:40.326213 18870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:40.330993 18870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:40.331421 18870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:40.333784 18870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:40.333876 18870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:40.333954 18870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:40.334136 18870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:34:40 dist-test-slave-drl0 krb5kdc[18710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163680, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:40.347030 18870 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:38213
I20260808 04:34:40.347167 19045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:38213 every 8 connection(s)
I20260808 04:34:40.348229 18870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:34:40.352161 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:40.335181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:42873 (local address 127.15.228.126:36681)
0808 04:34:40.335382 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:40.335387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:40.336626 (+ 1239us) server_negotiation.cc:407] Connection header received
0808 04:34:40.338256 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:40.338262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:40.338345 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:40.338456 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:40.339869 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.340685 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.342081 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.342389 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.342509 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:40.346439 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:40.346445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:40.346447 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:40.346495 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:40.348786 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:40.349796 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:40.349800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:40.349802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:40.349940 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:40.350529 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:40.350545 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:40.350547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:40.350812 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:40.350913 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:40.351767 (+ 854us) server_negotiation.cc:299] Negotiation successful
0808 04:34:40.351938 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 04:34:40.353268 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:40.335647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36681 (local address 127.15.228.65:42873)
0808 04:34:40.336374 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:40.336440 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:40.337966 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:40.338668 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:40.338675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:40.339143 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:40.339694 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:40.339709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.340837 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.340842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:40.341948 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:40.341964 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.342579 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.342588 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:40.342860 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:40.343716 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:40.343737 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:40.346094 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:40.348954 (+ 2860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:40.348959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:40.348975 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:40.349381 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:40.350097 (+ 716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:40.350106 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:40.350108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:40.350317 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:40.351068 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:40.351076 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:40.351411 (+ 335us) client_negotiation.cc:769] Sending connection context
0808 04:34:40.351620 (+ 209us) client_negotiation.cc:240] Negotiation successful
0808 04:34:40.352015 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":292,"threads_started":1}
I20260808 04:34:40.354481 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 18870
I20260808 04:34:40.354656 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:34:40.355014 19046 heartbeater.cc:347] Connected to a master server at 127.15.228.126:36681
I20260808 04:34:40.355103 16273 external_mini_cluster.cc:1428] Setting key bac94d62befb3531503fd83cc0c48911
I20260808 04:34:40.355316 19046 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:40.356782 19046 heartbeater.cc:511] Master 127.15.228.126:36681 requested a full tablet report, sending...
I20260808 04:34:40.358536 18766 ts_manager.cc:194] Registered new tserver with Master: bf60ee870f894a13beef385cf9a3d420 (127.15.228.65:38213)
I20260808 04:34:40.360230 18766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:42873
WARNING: no policy specified for kudu/127.15.228.66@KRBTEST.COM; defaulting to no policy
I20260808 04:34:40.367369 19060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:40.357726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36681 (local address 127.15.228.65:57251)
0808 04:34:40.358019 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:40.358036 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:40.358129 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:40.359555 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:40.359558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:40.359775 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:40.359895 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:40.359901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.360837 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.360844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:40.361954 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:40.361971 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.362796 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.362811 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:40.362904 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:40.363507 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:40.363529 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:40.363892 (+ 363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:40.365677 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:40.365681 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:40.365684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:40.365903 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:40.366191 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:40.366195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:40.366197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:40.366290 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:40.366721 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:40.366724 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:40.366837 (+ 113us) client_negotiation.cc:769] Sending connection context
0808 04:34:40.366987 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 04:34:40.367148 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":1,"thread_start_us":99,"threads_started":1}
I20260808 04:34:40.367362 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:40.359169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:57251 (local address 127.15.228.126:36681)
0808 04:34:40.359324 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:40.359329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:40.359343 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:34:40.359379 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:40.359381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:40.359431 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:40.359533 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:40.360020 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.360701 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.362111 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.362549 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.362663 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:40.364043 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:40.364047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:40.364051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:40.364076 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:40.365556 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:40.366008 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:40.366013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:40.366015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:40.366096 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:40.366387 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:40.366392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:40.366393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:40.366622 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:40.366733 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:40.367036 (+ 303us) server_negotiation.cc:299] Negotiation successful
0808 04:34:40.367148 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":66}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
I20260808 04:34:40.392304 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:36681
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:40.525205 19065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:40.525490 19065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:40.525543 19065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:40.531234 19065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:40.531316 19065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:40.531368 19065 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 04:34:40.531440 19065 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:34:40.531519 19065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:34:40.537968 19065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.228.126:36681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:40.539139 19065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:40.540396 19065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:40.554109 19075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:40.554241 19077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:40.554111 19074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:40.554724 19065 server_base.cc:1034] running on GCE node
I20260808 04:34:40.555323 19065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:40.556044 19065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:40.557193 19065 hybrid_clock.cc:648] HybridClock initialized: now 1786163680557158 us; error 50 us; skew 500 ppm
Aug 08 04:34:40 dist-test-slave-drl0 krb5kdc[18710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163680, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:40.560591 19065 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:34:40.561671 19065 webserver.cc:460] Webserver started at http://127.15.228.66:36043/ using document root <none> and password file <none>
I20260808 04:34:40.562572 19065 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:40.562693 19065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:40.563038 19065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:40.565579 19065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "17e39b4aa5ec48008e4477b502dda3b1"
format_stamp: "Formatted at 2026-08-08 04:34:40 on dist-test-slave-drl0"
server_key: "93c7bfd5e06131bd35642f963210773e"
server_key_iv: "dfeaa1a59a8c62d5d5499ca5076f4335"
server_key_version: "encryptionkey@0"
I20260808 04:34:40.566586 19065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "17e39b4aa5ec48008e4477b502dda3b1"
format_stamp: "Formatted at 2026-08-08 04:34:40 on dist-test-slave-drl0"
server_key: "93c7bfd5e06131bd35642f963210773e"
server_key_iv: "dfeaa1a59a8c62d5d5499ca5076f4335"
server_key_version: "encryptionkey@0"
I20260808 04:34:40.572513 19065 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260808 04:34:40.575739 19087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:40.577207 19065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:34:40.577340 19065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "17e39b4aa5ec48008e4477b502dda3b1"
format_stamp: "Formatted at 2026-08-08 04:34:40 on dist-test-slave-drl0"
server_key: "93c7bfd5e06131bd35642f963210773e"
server_key_iv: "dfeaa1a59a8c62d5d5499ca5076f4335"
server_key_version: "encryptionkey@0"
I20260808 04:34:40.577523 19065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:40.597771 19065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:40.602685 19065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:40.603011 19065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:40.605051 19065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:40.605165 19065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:40.605243 19065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:40.605291 19065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:34:40 dist-test-slave-drl0 krb5kdc[18710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163680, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:40.619280 19065 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:35549
I20260808 04:34:40.619356 19236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:35549 every 8 connection(s)
I20260808 04:34:40.620537 19065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:34:40.622535 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 19065
I20260808 04:34:40.622685 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:34:40.623086 16273 external_mini_cluster.cc:1428] Setting key b9ed95ffca4b1b971f4e05bc183a5d14
I20260808 04:34:40.624198 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:40.606772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:44329 (local address 127.15.228.126:36681)
0808 04:34:40.607015 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:40.607019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:40.608742 (+ 1723us) server_negotiation.cc:407] Connection header received
0808 04:34:40.610213 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:40.610216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:40.610301 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:40.610435 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:40.612749 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.613774 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.615365 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.615637 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.615772 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:40.619331 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:40.619336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:40.619339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:40.619384 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:40.621163 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:40.621787 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:40.621791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:40.621792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:40.621885 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:40.622410 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:40.622416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:40.622417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:40.622860 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:40.623109 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:40.623862 (+ 753us) server_negotiation.cc:299] Negotiation successful
0808 04:34:40.624045 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:34:40.625237 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:40.607469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36681 (local address 127.15.228.66:44329)
0808 04:34:40.608346 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:40.608560 (+ 214us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:40.609904 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:40.610692 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:40.610706 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:40.611494 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:40.612466 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:40.612499 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.613983 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.613987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:40.615180 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:40.615195 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.615841 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.615845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:40.616062 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:40.616966 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:40.616996 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:40.619027 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:40.621328 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:40.621333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:40.621349 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:40.621644 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:40.622016 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:40.622019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:40.622020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:40.622160 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:40.623125 (+ 965us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:40.623133 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:40.623516 (+ 383us) client_negotiation.cc:769] Sending connection context
0808 04:34:40.623750 (+ 234us) client_negotiation.cc:240] Negotiation successful
0808 04:34:40.624191 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":188,"threads_started":1}
I20260808 04:34:40.627272 19237 heartbeater.cc:347] Connected to a master server at 127.15.228.126:36681
I20260808 04:34:40.627663 19237 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:40.628409 19237 heartbeater.cc:511] Master 127.15.228.126:36681 requested a full tablet report, sending...
I20260808 04:34:40.630080 18764 ts_manager.cc:194] Registered new tserver with Master: 17e39b4aa5ec48008e4477b502dda3b1 (127.15.228.66:35549)
I20260808 04:34:40.631286 18764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:44329
WARNING: no policy specified for kudu/127.15.228.67@KRBTEST.COM; defaulting to no policy
I20260808 04:34:40.638761 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:40.630772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36681 (local address 127.15.228.66:55891)
0808 04:34:40.631335 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:40.631355 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:40.631566 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:40.632039 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:40.632042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:40.632301 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:40.632434 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:40.632440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.633232 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.633236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:40.633825 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:40.633833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.634312 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.634316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:40.634422 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:40.635012 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:40.635025 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:40.635354 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:40.637126 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:40.637130 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:40.637132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:40.637343 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:40.637634 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:40.637637 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:40.637639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:40.637727 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:40.638171 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:40.638175 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:40.638279 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 04:34:40.638404 (+ 125us) client_negotiation.cc:240] Negotiation successful
0808 04:34:40.638561 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":4,"thread_start_us":297,"threads_started":1}
I20260808 04:34:40.638746 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:40.631706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:55891 (local address 127.15.228.126:36681)
0808 04:34:40.631834 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:40.631838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:40.631849 (+ 11us) server_negotiation.cc:407] Connection header received
0808 04:34:40.631882 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:40.631883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:40.631919 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:40.632017 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:40.632557 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.633099 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.633951 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.634173 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.634283 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:40.635507 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:40.635510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:40.635513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:40.635539 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:40.636972 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:40.637448 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:40.637453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:40.637455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:40.637536 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:40.637832 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:40.637835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:40.637837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:40.638065 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:40.638221 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:40.638492 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 04:34:40.638601 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
I20260808 04:34:40.659663 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:36681
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:40.805583 19249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:40.805876 19249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:40.805933 19249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:40.810621 19249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:40.810693 19249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:40.810736 19249 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 04:34:40.810751 19249 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:34:40.810784 19249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:34:40.819278 19249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.228.126:36681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:40.820500 19249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:40.821625 19249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:40.834713 19263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:40.834735 19261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:40.834730 19255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:40.835201 19249 server_base.cc:1034] running on GCE node
I20260808 04:34:40.835692 19249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:40.836378 19249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:40.837622 19249 hybrid_clock.cc:648] HybridClock initialized: now 1786163680837580 us; error 61 us; skew 500 ppm
Aug 08 04:34:40 dist-test-slave-drl0 krb5kdc[18710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163680, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:40.841078 19249 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:34:40.842150 19249 webserver.cc:460] Webserver started at http://127.15.228.67:36533/ using document root <none> and password file <none>
I20260808 04:34:40.842759 19249 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:40.842851 19249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:40.843130 19249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:40.845273 19249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "352419849eb14ccea05ab379f0d48140"
format_stamp: "Formatted at 2026-08-08 04:34:40 on dist-test-slave-drl0"
server_key: "7941cc6aa645248154f7e572fca70a96"
server_key_iv: "61fc0feb4544c206723be190b5f2030a"
server_key_version: "encryptionkey@0"
I20260808 04:34:40.846235 19249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "352419849eb14ccea05ab379f0d48140"
format_stamp: "Formatted at 2026-08-08 04:34:40 on dist-test-slave-drl0"
server_key: "7941cc6aa645248154f7e572fca70a96"
server_key_iv: "61fc0feb4544c206723be190b5f2030a"
server_key_version: "encryptionkey@0"
I20260808 04:34:40.853044 19249 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260808 04:34:40.857847 19273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:40.860055 19249 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 04:34:40.860256 19249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "352419849eb14ccea05ab379f0d48140"
format_stamp: "Formatted at 2026-08-08 04:34:40 on dist-test-slave-drl0"
server_key: "7941cc6aa645248154f7e572fca70a96"
server_key_iv: "61fc0feb4544c206723be190b5f2030a"
server_key_version: "encryptionkey@0"
I20260808 04:34:40.860428 19249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:40.879390 19249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:40.883224 19249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:40.883469 19249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:40.885603 19249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:40.885666 19249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:40.885761 19249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:40.885797 19249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:34:40 dist-test-slave-drl0 krb5kdc[18710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163680, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:40.900642 19249 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:40013
I20260808 04:34:40.900677 19424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:40013 every 8 connection(s)
I20260808 04:34:40.901865 19249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:34:40.904940 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:40.886986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:47925 (local address 127.15.228.126:36681)
0808 04:34:40.887129 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:40.887133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:40.888276 (+ 1143us) server_negotiation.cc:407] Connection header received
0808 04:34:40.889946 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:40.889949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:40.890010 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:40.890128 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:40.891949 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.892828 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.895599 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.896111 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.896282 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:40.900532 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:40.900537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:40.900541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:40.900575 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:40.902222 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:40.903188 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:40.903194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:40.903196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:40.903405 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:40.903805 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:40.903808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:40.903810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:40.904022 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:40.904107 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:40.904688 (+ 581us) server_negotiation.cc:299] Negotiation successful
0808 04:34:40.904809 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:34:40.905989 19282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:40.887525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36681 (local address 127.15.228.67:47925)
0808 04:34:40.888102 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:40.888146 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:40.889692 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:40.890313 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:40.890320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:40.891183 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:40.891787 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:40.891802 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.892975 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.892979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:40.895331 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:40.895366 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.896357 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.896363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:40.896661 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:40.898280 (+ 1619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:40.898310 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:40.900246 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:40.902426 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:40.902434 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:40.902450 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:40.903047 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:40.903533 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:40.903536 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:40.903538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:40.903694 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:40.904161 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:40.904166 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:40.904417 (+ 251us) client_negotiation.cc:769] Sending connection context
0808 04:34:40.904572 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 04:34:40.904840 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":162,"threads_started":1}
I20260808 04:34:40.908018 19425 heartbeater.cc:347] Connected to a master server at 127.15.228.126:36681
I20260808 04:34:40.908311 19425 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:40.909276 19425 heartbeater.cc:511] Master 127.15.228.126:36681 requested a full tablet report, sending...
I20260808 04:34:40.909612 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 19249
I20260808 04:34:40.909758 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:34:40.910135 16273 external_mini_cluster.cc:1428] Setting key 536be6408c6f0eab7eddcf58d68d20bc
I20260808 04:34:40.911079 18764 ts_manager.cc:194] Registered new tserver with Master: 352419849eb14ccea05ab379f0d48140 (127.15.228.67:40013)
I20260808 04:34:40.912263 18764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:47925
I20260808 04:34:40.913203 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:34:40.922047 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:40.911522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36681 (local address 127.15.228.67:57907)
0808 04:34:40.911916 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:40.911935 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:40.912071 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:40.913390 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:40.913394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:40.913616 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:40.913745 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:40.913752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.914789 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.914794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:40.915593 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:40.915606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.916210 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.916214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:40.916333 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:40.917175 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:40.917189 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:40.917631 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:40.919798 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:40.919802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:40.919805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:40.920076 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:40.920656 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:40.920660 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:40.920663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:40.920788 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:40.921326 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:40.921329 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:40.921453 (+ 124us) client_negotiation.cc:769] Sending connection context
0808 04:34:40.921592 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 04:34:40.921822 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":148,"threads_started":1}
I20260808 04:34:40.922031 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:40.912874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:57907 (local address 127.15.228.126:36681)
0808 04:34:40.913002 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:40.913007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:40.913023 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:34:40.913075 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:40.913078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:40.913123 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:40.913212 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:40.913878 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.914559 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.915756 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.916023 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.916202 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:40.917822 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:40.917826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:40.917828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:40.917854 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:40.919629 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:40.920217 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:40.920222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:40.920224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:40.920464 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:40.920907 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:40.920913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:40.920915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:40.921207 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:40.921341 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:40.921662 (+ 321us) server_negotiation.cc:299] Negotiation successful
0808 04:34:40.921817 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Aug 08 04:34:40 dist-test-slave-drl0 krb5kdc[18710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 04:34:40 dist-test-slave-drl0 krb5kdc[18710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:40.952523 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:40.940378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60594 (local address 127.15.228.126:36681)
0808 04:34:40.940500 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:40.940503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:40.940515 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:34:40.940558 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:40.940560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:40.940600 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:40.940664 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:40.941769 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.942463 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.943601 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.943829 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.943919 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:40.947364 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:40.947371 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:40.947375 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:40.947428 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:40.949590 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:40.950676 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:40.950680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:40.950681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:40.950788 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:40.951330 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:40.951407 (+ 77us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:40.951408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:40.951645 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:40.951907 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:40.952231 (+ 324us) server_negotiation.cc:299] Negotiation successful
0808 04:34:40.952370 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 04:34:40.956832 18766 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:34:40.960413 18766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:34:40.980487 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:40.972890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:35549 (local address 127.0.0.1:39728)
0808 04:34:40.974657 (+ 1767us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:40.974676 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:40.975057 (+ 381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:40.975851 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:40.975853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:40.975878 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:40.976025 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:40.976030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.977608 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.977610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:40.978608 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:40.978618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.980004 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.980008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:40.980116 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:40.980141 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:34:40.980209 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 04:34:40.980278 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1676,"mutex_wait_us":131,"spinlock_wait_cycles":2944,"thread_start_us":159,"threads_started":1}
I20260808 04:34:40.980892 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:40.974056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49538 (local address 127.15.228.65:38213)
0808 04:34:40.974489 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:40.974496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:40.974530 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:34:40.974874 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:40.974883 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:40.975237 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:40.975427 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:40.975827 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.977054 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.979085 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.979829 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.979949 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:40.980091 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:40.980576 (+ 485us) server_negotiation.cc:299] Negotiation successful
0808 04:34:40.980704 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":107,"threads_started":1}
I20260808 04:34:40.981091 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:40.973070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39728 (local address 127.15.228.66:35549)
0808 04:34:40.973814 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:40.973839 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:40.974775 (+ 936us) server_negotiation.cc:407] Connection header received
0808 04:34:40.975253 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:40.975259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:40.975522 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:40.975693 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:40.976144 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.977473 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.978746 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.979638 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.979748 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:40.979936 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:40.980704 (+ 768us) server_negotiation.cc:299] Negotiation successful
0808 04:34:40.980839 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":153,"threads_started":1}
I20260808 04:34:40.981194 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:40.972890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:38213 (local address 127.0.0.1:49538)
0808 04:34:40.974023 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:40.974088 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:40.974262 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:40.975432 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:40.975439 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:40.975463 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:40.975644 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:40.975652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.977208 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.977212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:40.978915 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:40.978949 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.980147 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.980151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:40.980251 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:40.980270 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 04:34:40.980329 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 04:34:40.980386 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":193,"spinlock_wait_cycles":14208,"thread_start_us":199,"threads_started":1}
I20260808 04:34:40.980356 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:40.973931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:40013 (local address 127.0.0.1:46904)
0808 04:34:40.974974 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:40.974990 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:40.975075 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:40.975841 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:40.975844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:40.975864 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:40.976021 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:40.976028 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.977391 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.977394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:40.978915 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:40.978953 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.979841 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.979847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:40.979950 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:40.979993 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 04:34:40.980094 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 04:34:40.980213 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":87,"thread_start_us":161,"threads_started":1}
I20260808 04:34:40.984160 19153 tablet_service.cc:1467] Processing CreateTablet for tablet 868eb07e00384b17a587eba358f4bf30 (DEFAULT_TABLE table=test-table [id=6d2ad075549b4cf3bbdc1dc154fe3496]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:40.984194 18948 tablet_service.cc:1467] Processing CreateTablet for tablet 868eb07e00384b17a587eba358f4bf30 (DEFAULT_TABLE table=test-table [id=6d2ad075549b4cf3bbdc1dc154fe3496]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:40.985242 18948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868eb07e00384b17a587eba358f4bf30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:40.985208 19153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868eb07e00384b17a587eba358f4bf30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:40.988248 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:40.974156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.15.228.67:40013)
0808 04:34:40.974820 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:40.974849 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:40.975091 (+ 242us) server_negotiation.cc:407] Connection header received
0808 04:34:40.975186 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:40.975191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:40.975369 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:40.975640 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:40.976181 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.977264 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.979109 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:40.979625 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:40.979757 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:40.987772 (+ 8015us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:40.987876 (+ 104us) server_negotiation.cc:299] Negotiation successful
0808 04:34:40.988044 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":158,"threads_started":1}
I20260808 04:34:40.991726 19331 tablet_service.cc:1467] Processing CreateTablet for tablet 868eb07e00384b17a587eba358f4bf30 (DEFAULT_TABLE table=test-table [id=6d2ad075549b4cf3bbdc1dc154fe3496]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:40.993438 19331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868eb07e00384b17a587eba358f4bf30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:40.998390 19457 tablet_bootstrap.cc:492] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420: Bootstrap starting.
I20260808 04:34:41.001391 19457 tablet_bootstrap.cc:654] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:41.002466 19457 log.cc:824] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:41.003407 19458 tablet_bootstrap.cc:492] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1: Bootstrap starting.
I20260808 04:34:41.004336 19460 tablet_bootstrap.cc:492] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140: Bootstrap starting.
I20260808 04:34:41.004932 19457 tablet_bootstrap.cc:492] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420: No bootstrap required, opened a new log
I20260808 04:34:41.005172 19457 ts_tablet_manager.cc:1397] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:34:41.006395 19460 tablet_bootstrap.cc:654] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:41.006412 19458 tablet_bootstrap.cc:654] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:41.007396 19460 log.cc:824] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:41.007404 19458 log.cc:824] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:41.009814 19458 tablet_bootstrap.cc:492] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1: No bootstrap required, opened a new log
I20260808 04:34:41.009661 19460 tablet_bootstrap.cc:492] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140: No bootstrap required, opened a new log
I20260808 04:34:41.010049 19458 ts_tablet_manager.cc:1397] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:34:41.010257 19460 ts_tablet_manager.cc:1397] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:34:41.011638 19457 raft_consensus.cc:348] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.012133 19457 raft_consensus.cc:374] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:41.012207 19457 raft_consensus.cc:729] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf60ee870f894a13beef385cf9a3d420, State: Initialized, Role: FOLLOWER
I20260808 04:34:41.012832 19457 consensus_queue.cc:260] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.013859 19046 heartbeater.cc:503] Master 127.15.228.126:36681 was elected leader, sending a full tablet report...
I20260808 04:34:41.014293 19457 ts_tablet_manager.cc:1428] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260808 04:34:41.015129 19460 raft_consensus.cc:348] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.015447 19460 raft_consensus.cc:374] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:41.015509 19460 raft_consensus.cc:729] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352419849eb14ccea05ab379f0d48140, State: Initialized, Role: FOLLOWER
I20260808 04:34:41.016057 19460 consensus_queue.cc:260] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.017056 19425 heartbeater.cc:503] Master 127.15.228.126:36681 was elected leader, sending a full tablet report...
I20260808 04:34:41.017452 19460 ts_tablet_manager.cc:1428] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260808 04:34:41.016738 19458 raft_consensus.cc:348] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.017910 19458 raft_consensus.cc:374] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:41.017967 19458 raft_consensus.cc:729] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17e39b4aa5ec48008e4477b502dda3b1, State: Initialized, Role: FOLLOWER
I20260808 04:34:41.018507 19458 consensus_queue.cc:260] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.019522 19237 heartbeater.cc:503] Master 127.15.228.126:36681 was elected leader, sending a full tablet report...
I20260808 04:34:41.019779 19458 ts_tablet_manager.cc:1428] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260808 04:34:41.041328 19463 raft_consensus.cc:482] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:34:41.041566 19463 raft_consensus.cc:504] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.043608 19463 leader_election.cc:288] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17e39b4aa5ec48008e4477b502dda3b1 (127.15.228.66:35549), 352419849eb14ccea05ab379f0d48140 (127.15.228.67:40013)
I20260808 04:34:41.049678 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:41.043559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:42135 (local address 127.15.228.66:35549)
0808 04:34:41.043708 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:41.043713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:41.043998 (+ 285us) server_negotiation.cc:407] Connection header received
0808 04:34:41.044144 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:41.044147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:41.044220 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:41.044320 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:41.044910 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.045786 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.048072 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.048769 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.048887 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:41.048949 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:41.049458 (+ 509us) server_negotiation.cc:299] Negotiation successful
0808 04:34:41.049537 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:34:41.049678 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:41.043449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:40621 (local address 127.15.228.67:40013)
0808 04:34:41.043720 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:41.043726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:41.044043 (+ 317us) server_negotiation.cc:407] Connection header received
0808 04:34:41.044136 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:41.044140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:41.044220 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:41.044320 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:41.044908 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.045786 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.048072 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.048734 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.048887 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:41.048962 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:41.049447 (+ 485us) server_negotiation.cc:299] Negotiation successful
0808 04:34:41.049532 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260808 04:34:41.049744 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:41.043328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:40013 (local address 127.15.228.65:40621)
0808 04:34:41.043838 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:41.043851 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:41.043930 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:41.044369 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:41.044372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:41.044419 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:41.044718 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:41.044728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.045942 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.045945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:41.047903 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:41.047916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.048920 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.048925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:41.049196 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:41.049309 (+ 113us) client_negotiation.cc:769] Sending connection context
0808 04:34:41.049415 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 04:34:41.049525 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":261,"spinlock_wait_cycles":33408,"thread_start_us":129,"threads_started":1}
I20260808 04:34:41.049965 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:41.043213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:35549 (local address 127.15.228.65:42135)
0808 04:34:41.043662 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:41.043678 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:41.043909 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:41.044369 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:41.044372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:41.044421 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:41.044718 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:41.044735 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.046053 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.046056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:41.047903 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:41.047916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.048920 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.048925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:41.049193 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:41.049309 (+ 116us) client_negotiation.cc:769] Sending connection context
0808 04:34:41.049401 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 04:34:41.049525 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":301,"spinlock_wait_cycles":11008,"thread_start_us":159,"threads_started":1}
I20260808 04:34:41.050516 19364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868eb07e00384b17a587eba358f4bf30" candidate_uuid: "bf60ee870f894a13beef385cf9a3d420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "352419849eb14ccea05ab379f0d48140" is_pre_election: true
I20260808 04:34:41.050662 19183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868eb07e00384b17a587eba358f4bf30" candidate_uuid: "bf60ee870f894a13beef385cf9a3d420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17e39b4aa5ec48008e4477b502dda3b1" is_pre_election: true
I20260808 04:34:41.050877 19364 raft_consensus.cc:2454] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf60ee870f894a13beef385cf9a3d420 in term 0.
I20260808 04:34:41.051020 19183 raft_consensus.cc:2454] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf60ee870f894a13beef385cf9a3d420 in term 0.
I20260808 04:34:41.051488 18890 leader_election.cc:302] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 352419849eb14ccea05ab379f0d48140, bf60ee870f894a13beef385cf9a3d420; no voters:
I20260808 04:34:41.051800 19463 raft_consensus.cc:2781] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:34:41.051896 19463 raft_consensus.cc:482] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:34:41.051942 19463 raft_consensus.cc:3037] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:41.053278 19463 raft_consensus.cc:504] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.054003 19463 leader_election.cc:288] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [CANDIDATE]: Term 1 election: Requested vote from peers 17e39b4aa5ec48008e4477b502dda3b1 (127.15.228.66:35549), 352419849eb14ccea05ab379f0d48140 (127.15.228.67:40013)
I20260808 04:34:41.054474 19183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868eb07e00384b17a587eba358f4bf30" candidate_uuid: "bf60ee870f894a13beef385cf9a3d420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17e39b4aa5ec48008e4477b502dda3b1"
I20260808 04:34:41.054574 19364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868eb07e00384b17a587eba358f4bf30" candidate_uuid: "bf60ee870f894a13beef385cf9a3d420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "352419849eb14ccea05ab379f0d48140"
I20260808 04:34:41.054746 19364 raft_consensus.cc:3037] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:41.054711 19183 raft_consensus.cc:3037] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:41.056053 19364 raft_consensus.cc:2454] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf60ee870f894a13beef385cf9a3d420 in term 1.
I20260808 04:34:41.056490 18890 leader_election.cc:302] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 352419849eb14ccea05ab379f0d48140, bf60ee870f894a13beef385cf9a3d420; no voters:
I20260808 04:34:41.056732 19463 raft_consensus.cc:2781] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:34:41.056874 19183 raft_consensus.cc:2454] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf60ee870f894a13beef385cf9a3d420 in term 1.
I20260808 04:34:41.057044 19463 raft_consensus.cc:686] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [term 1 LEADER]: Becoming Leader. State: Replica: bf60ee870f894a13beef385cf9a3d420, State: Running, Role: LEADER
I20260808 04:34:41.057468 19463 consensus_queue.cc:237] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.061393 18766 catalog_manager.cc:5495] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf60ee870f894a13beef385cf9a3d420 (127.15.228.65). New cstate: current_term: 1 leader_uuid: "bf60ee870f894a13beef385cf9a3d420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } health_report { overall_health: UNKNOWN } } }
W20260808 04:34:41.100334 19048 tablet.cc:2369] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 04:34:41 dist-test-slave-drl0 krb5kdc[18710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163679, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:41.120123 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:41.109242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36681 (local address 127.0.0.1:60598)
0808 04:34:41.109811 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:41.109830 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:41.110032 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:41.110490 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:41.110493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:41.110986 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:41.111129 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:41.111137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.112440 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.112445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:41.113251 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:41.113263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.113846 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.113854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:41.114021 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:41.114698 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:41.114711 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:41.116289 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:41.118434 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:41.118444 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:41.118450 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:41.118681 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:41.119068 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:41.119070 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:41.119071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:41.119152 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:41.119615 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:41.119622 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:41.119731 (+ 109us) client_negotiation.cc:769] Sending connection context
0808 04:34:41.119840 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 04:34:41.119974 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":230,"threads_started":1}
I20260808 04:34:41.120229 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:41.109447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60598 (local address 127.15.228.126:36681)
0808 04:34:41.109640 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:41.109646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:41.109978 (+ 332us) server_negotiation.cc:407] Connection header received
0808 04:34:41.110224 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:41.110228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:41.110280 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:41.110375 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:41.111548 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.112190 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.113405 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.113627 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.113721 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:41.116456 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:41.116459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:41.116461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:41.116484 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:41.118283 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:41.118826 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:41.118829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:41.118830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:41.118909 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:41.119259 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:41.119262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:41.119263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:41.119459 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:41.119545 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:41.119957 (+ 412us) server_negotiation.cc:299] Negotiation successful
0808 04:34:41.120073 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260808 04:34:41.122130 19238 tablet.cc:2369] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:34:41.124863 18766 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:60598:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 04:34:41.125998 18766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:34:41.133610 19153 tablet_service.cc:1467] Processing CreateTablet for tablet f012b03a86be4533a6e69f83808e9be9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec57c38796ed40e3a507efc276960f4f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:34:41.133855 19331 tablet_service.cc:1467] Processing CreateTablet for tablet f012b03a86be4533a6e69f83808e9be9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec57c38796ed40e3a507efc276960f4f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:34:41.133636 18948 tablet_service.cc:1467] Processing CreateTablet for tablet f012b03a86be4533a6e69f83808e9be9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec57c38796ed40e3a507efc276960f4f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:34:41.134613 18948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f012b03a86be4533a6e69f83808e9be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:41.134613 19331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f012b03a86be4533a6e69f83808e9be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:41.134619 19153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f012b03a86be4533a6e69f83808e9be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:41.139017 19457 tablet_bootstrap.cc:492] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420: Bootstrap starting.
I20260808 04:34:41.139487 19458 tablet_bootstrap.cc:492] T f012b03a86be4533a6e69f83808e9be9 P 17e39b4aa5ec48008e4477b502dda3b1: Bootstrap starting.
I20260808 04:34:41.140817 19457 tablet_bootstrap.cc:654] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:41.140962 19458 tablet_bootstrap.cc:654] T f012b03a86be4533a6e69f83808e9be9 P 17e39b4aa5ec48008e4477b502dda3b1: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:41.141607 19460 tablet_bootstrap.cc:492] T f012b03a86be4533a6e69f83808e9be9 P 352419849eb14ccea05ab379f0d48140: Bootstrap starting.
I20260808 04:34:41.142639 19457 tablet_bootstrap.cc:492] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420: No bootstrap required, opened a new log
I20260808 04:34:41.142643 19458 tablet_bootstrap.cc:492] T f012b03a86be4533a6e69f83808e9be9 P 17e39b4aa5ec48008e4477b502dda3b1: No bootstrap required, opened a new log
I20260808 04:34:41.142745 19458 ts_tablet_manager.cc:1397] T f012b03a86be4533a6e69f83808e9be9 P 17e39b4aa5ec48008e4477b502dda3b1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:34:41.142745 19457 ts_tablet_manager.cc:1397] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260808 04:34:41.142750 19460 tablet_bootstrap.cc:654] T f012b03a86be4533a6e69f83808e9be9 P 352419849eb14ccea05ab379f0d48140: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:41.143478 19457 raft_consensus.cc:348] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.143630 19457 raft_consensus.cc:374] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:41.143697 19457 raft_consensus.cc:729] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf60ee870f894a13beef385cf9a3d420, State: Initialized, Role: FOLLOWER
I20260808 04:34:41.143855 19457 consensus_queue.cc:260] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.143841 19458 raft_consensus.cc:348] T f012b03a86be4533a6e69f83808e9be9 P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.143958 19458 raft_consensus.cc:374] T f012b03a86be4533a6e69f83808e9be9 P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:41.143994 19458 raft_consensus.cc:729] T f012b03a86be4533a6e69f83808e9be9 P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17e39b4aa5ec48008e4477b502dda3b1, State: Initialized, Role: FOLLOWER
I20260808 04:34:41.144131 19458 consensus_queue.cc:260] T f012b03a86be4533a6e69f83808e9be9 P 17e39b4aa5ec48008e4477b502dda3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.144142 19463 tablet_replica.cc:439] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } } }
I20260808 04:34:41.144264 19460 tablet_bootstrap.cc:492] T f012b03a86be4533a6e69f83808e9be9 P 352419849eb14ccea05ab379f0d48140: No bootstrap required, opened a new log
I20260808 04:34:41.144272 19463 tablet_replica.cc:442] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:34:41.144361 19460 ts_tablet_manager.cc:1397] T f012b03a86be4533a6e69f83808e9be9 P 352419849eb14ccea05ab379f0d48140: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20260808 04:34:41.144421 19466 tablet_replica.cc:439] T f012b03a86be4533a6e69f83808e9be9 P 17e39b4aa5ec48008e4477b502dda3b1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } } }
I20260808 04:34:41.144565 19466 tablet_replica.cc:442] T f012b03a86be4533a6e69f83808e9be9 P 17e39b4aa5ec48008e4477b502dda3b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:34:41.144840 19460 raft_consensus.cc:348] T f012b03a86be4533a6e69f83808e9be9 P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.144965 19460 raft_consensus.cc:374] T f012b03a86be4533a6e69f83808e9be9 P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:41.144980 19457 ts_tablet_manager.cc:1428] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 04:34:41.145006 19460 raft_consensus.cc:729] T f012b03a86be4533a6e69f83808e9be9 P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352419849eb14ccea05ab379f0d48140, State: Initialized, Role: FOLLOWER
I20260808 04:34:41.145205 19460 consensus_queue.cc:260] T f012b03a86be4533a6e69f83808e9be9 P 352419849eb14ccea05ab379f0d48140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.145501 19464 tablet_replica.cc:439] T f012b03a86be4533a6e69f83808e9be9 P 352419849eb14ccea05ab379f0d48140: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } } }
I20260808 04:34:41.145627 19464 tablet_replica.cc:442] T f012b03a86be4533a6e69f83808e9be9 P 352419849eb14ccea05ab379f0d48140: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:34:41.145892 19460 ts_tablet_manager.cc:1428] T f012b03a86be4533a6e69f83808e9be9 P 352419849eb14ccea05ab379f0d48140: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 04:34:41.144436 19458 ts_tablet_manager.cc:1428] T f012b03a86be4533a6e69f83808e9be9 P 17e39b4aa5ec48008e4477b502dda3b1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20260808 04:34:41.154172 19428 tablet.cc:2369] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:34:41.164338 19463 raft_consensus.cc:482] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:34:41.164489 19463 raft_consensus.cc:504] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.165030 19463 leader_election.cc:288] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17e39b4aa5ec48008e4477b502dda3b1 (127.15.228.66:35549), 352419849eb14ccea05ab379f0d48140 (127.15.228.67:40013)
I20260808 04:34:41.165494 19183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f012b03a86be4533a6e69f83808e9be9" candidate_uuid: "bf60ee870f894a13beef385cf9a3d420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17e39b4aa5ec48008e4477b502dda3b1" is_pre_election: true
I20260808 04:34:41.165508 19364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f012b03a86be4533a6e69f83808e9be9" candidate_uuid: "bf60ee870f894a13beef385cf9a3d420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "352419849eb14ccea05ab379f0d48140" is_pre_election: true
I20260808 04:34:41.165671 19183 raft_consensus.cc:2454] T f012b03a86be4533a6e69f83808e9be9 P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf60ee870f894a13beef385cf9a3d420 in term 0.
I20260808 04:34:41.165671 19364 raft_consensus.cc:2454] T f012b03a86be4533a6e69f83808e9be9 P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf60ee870f894a13beef385cf9a3d420 in term 0.
I20260808 04:34:41.165948 18890 leader_election.cc:302] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17e39b4aa5ec48008e4477b502dda3b1, 352419849eb14ccea05ab379f0d48140, bf60ee870f894a13beef385cf9a3d420; no voters:
I20260808 04:34:41.166198 19463 raft_consensus.cc:2781] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:34:41.166288 19463 raft_consensus.cc:482] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:34:41.166345 19463 raft_consensus.cc:3037] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:41.167583 19463 raft_consensus.cc:504] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.167937 19463 leader_election.cc:288] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420 [CANDIDATE]: Term 1 election: Requested vote from peers 17e39b4aa5ec48008e4477b502dda3b1 (127.15.228.66:35549), 352419849eb14ccea05ab379f0d48140 (127.15.228.67:40013)
I20260808 04:34:41.168865 19364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f012b03a86be4533a6e69f83808e9be9" candidate_uuid: "bf60ee870f894a13beef385cf9a3d420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "352419849eb14ccea05ab379f0d48140"
I20260808 04:34:41.168865 19183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f012b03a86be4533a6e69f83808e9be9" candidate_uuid: "bf60ee870f894a13beef385cf9a3d420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17e39b4aa5ec48008e4477b502dda3b1"
I20260808 04:34:41.169094 19183 raft_consensus.cc:3037] T f012b03a86be4533a6e69f83808e9be9 P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:41.169094 19364 raft_consensus.cc:3037] T f012b03a86be4533a6e69f83808e9be9 P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:41.170660 19364 raft_consensus.cc:2454] T f012b03a86be4533a6e69f83808e9be9 P 352419849eb14ccea05ab379f0d48140 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf60ee870f894a13beef385cf9a3d420 in term 1.
I20260808 04:34:41.170646 19183 raft_consensus.cc:2454] T f012b03a86be4533a6e69f83808e9be9 P 17e39b4aa5ec48008e4477b502dda3b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf60ee870f894a13beef385cf9a3d420 in term 1.
I20260808 04:34:41.171191 18890 leader_election.cc:302] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17e39b4aa5ec48008e4477b502dda3b1, 352419849eb14ccea05ab379f0d48140, bf60ee870f894a13beef385cf9a3d420; no voters:
I20260808 04:34:41.171482 19463 raft_consensus.cc:2781] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:34:41.171644 19463 raft_consensus.cc:686] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420 [term 1 LEADER]: Becoming Leader. State: Replica: bf60ee870f894a13beef385cf9a3d420, State: Running, Role: LEADER
I20260808 04:34:41.171844 19463 consensus_queue.cc:237] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.173244 19469 tablet_replica.cc:439] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420: TxnStatusTablet state changed. Reason: New leader bf60ee870f894a13beef385cf9a3d420. Latest consensus state: current_term: 1 leader_uuid: "bf60ee870f894a13beef385cf9a3d420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } } }
I20260808 04:34:41.173385 19469 tablet_replica.cc:442] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:34:41.173758 19492 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 04:34:41.174232 19492 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 04:34:41.174609 18764 catalog_manager.cc:5495] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf60ee870f894a13beef385cf9a3d420 (127.15.228.65). New cstate: current_term: 1 leader_uuid: "bf60ee870f894a13beef385cf9a3d420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } health_report { overall_health: UNKNOWN } } }
I20260808 04:34:41.192791 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:41.188543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49550 (local address 127.15.228.65:38213)
0808 04:34:41.188764 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:41.188769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:41.188784 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:34:41.188852 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:41.188855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:41.188922 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:41.189042 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:34:41.189329 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.189913 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.191007 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.191357 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.191519 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:41.191869 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:34:41.192326 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:34:41.192481 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:41.192563 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 04:34:41.192625 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260808 04:34:41.192790 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:41.188235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:38213 (local address 127.0.0.1:49550)
0808 04:34:41.188424 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:41.188444 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:41.188547 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:41.189055 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:41.189058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:41.189067 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:34:41.189208 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:41.189215 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.190040 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.190044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:41.190832 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:41.190843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.191519 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.191523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:41.191638 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:41.191689 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:34:41.192452 (+ 763us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:34:41.192470 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 04:34:41.192550 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 04:34:41.192647 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 04:34:41.200596 19364 raft_consensus.cc:1261] T f012b03a86be4533a6e69f83808e9be9 P 352419849eb14ccea05ab379f0d48140 [term 1 FOLLOWER]: Refusing update from remote peer bf60ee870f894a13beef385cf9a3d420: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:34:41.200615 19183 raft_consensus.cc:1261] T f012b03a86be4533a6e69f83808e9be9 P 17e39b4aa5ec48008e4477b502dda3b1 [term 1 FOLLOWER]: Refusing update from remote peer bf60ee870f894a13beef385cf9a3d420: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:34:41.201570 19463 consensus_queue.cc:1035] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:34:41.201864 19469 consensus_queue.cc:1035] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:34:41.206351 19464 tablet_replica.cc:439] T f012b03a86be4533a6e69f83808e9be9 P 352419849eb14ccea05ab379f0d48140: TxnStatusTablet state changed. Reason: New leader bf60ee870f894a13beef385cf9a3d420. Latest consensus state: current_term: 1 leader_uuid: "bf60ee870f894a13beef385cf9a3d420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } } }
I20260808 04:34:41.206516 19464 tablet_replica.cc:442] T f012b03a86be4533a6e69f83808e9be9 P 352419849eb14ccea05ab379f0d48140: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:34:41.207120 19466 tablet_replica.cc:439] T f012b03a86be4533a6e69f83808e9be9 P 17e39b4aa5ec48008e4477b502dda3b1: TxnStatusTablet state changed. Reason: New leader bf60ee870f894a13beef385cf9a3d420. Latest consensus state: current_term: 1 leader_uuid: "bf60ee870f894a13beef385cf9a3d420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } } }
I20260808 04:34:41.207257 19466 tablet_replica.cc:442] T f012b03a86be4533a6e69f83808e9be9 P 17e39b4aa5ec48008e4477b502dda3b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:34:41.211145 19497 mvcc.cc:205] Tried to move back new op lower bound from 7316126438187180032 to 7316126438081859584. Current Snapshot: MvccSnapshot[applied={T|T < 7316126438187180032}]
I20260808 04:34:41.211388 19463 tablet_replica.cc:439] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf60ee870f894a13beef385cf9a3d420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } } }
I20260808 04:34:41.211529 19463 tablet_replica.cc:442] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:34:41.212643 19469 tablet_replica.cc:439] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf60ee870f894a13beef385cf9a3d420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } } }
I20260808 04:34:41.213109 19469 tablet_replica.cc:442] T f012b03a86be4533a6e69f83808e9be9 P bf60ee870f894a13beef385cf9a3d420: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:34:41.213439 19499 mvcc.cc:205] Tried to move back new op lower bound from 7316126438187180032 to 7316126438081859584. Current Snapshot: MvccSnapshot[applied={T|T < 7316126438187180032}]
I20260808 04:34:41.214336 19498 mvcc.cc:205] Tried to move back new op lower bound from 7316126438187180032 to 7316126438081859584. Current Snapshot: MvccSnapshot[applied={T|T < 7316126438187180032}]
I20260808 04:34:41.214555 19464 tablet_replica.cc:439] T f012b03a86be4533a6e69f83808e9be9 P 352419849eb14ccea05ab379f0d48140: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf60ee870f894a13beef385cf9a3d420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } } }
I20260808 04:34:41.214782 19464 tablet_replica.cc:442] T f012b03a86be4533a6e69f83808e9be9 P 352419849eb14ccea05ab379f0d48140: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:34:41.216146 19466 tablet_replica.cc:439] T f012b03a86be4533a6e69f83808e9be9 P 17e39b4aa5ec48008e4477b502dda3b1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf60ee870f894a13beef385cf9a3d420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } } }
I20260808 04:34:41.216281 19466 tablet_replica.cc:442] T f012b03a86be4533a6e69f83808e9be9 P 17e39b4aa5ec48008e4477b502dda3b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:34:41.224392 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:41.220430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49564 (local address 127.15.228.65:38213)
0808 04:34:41.220544 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:41.220547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:41.220561 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:34:41.220672 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:41.220674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:41.220717 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:41.220778 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:34:41.221126 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.221716 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.223114 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.223360 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.223446 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:41.223800 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:34:41.223882 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:34:41.223943 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:41.224073 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 04:34:41.224123 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 04:34:41.245103 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:41.240771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:60207 (local address 127.15.228.65:38213)
0808 04:34:41.240916 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:41.240929 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:41.241145 (+ 216us) server_negotiation.cc:407] Connection header received
0808 04:34:41.241267 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:41.241270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:41.241325 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:41.241412 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:34:41.241683 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.242512 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.243615 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.243968 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.244078 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:41.244557 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:34:41.244654 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:34:41.244739 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:41.244907 (+ 168us) server_negotiation.cc:299] Negotiation successful
0808 04:34:41.244964 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:34:41.245108 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:41.240706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:38213 (local address 127.15.228.65:60207)
0808 04:34:41.241058 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:41.241071 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:41.241175 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:41.241455 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:41.241458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:41.241464 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:34:41.241577 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:41.241581 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.242810 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.242814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:41.243496 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:41.243506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.244153 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.244156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:41.244253 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:41.244307 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:34:41.244783 (+ 476us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:34:41.244791 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 04:34:41.244862 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 04:34:41.244942 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":146,"threads_started":1}
I20260808 04:34:41.259290 19364 raft_consensus.cc:1261] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140 [term 1 FOLLOWER]: Refusing update from remote peer bf60ee870f894a13beef385cf9a3d420: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:34:41.259524 19183 raft_consensus.cc:1261] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1 [term 1 FOLLOWER]: Refusing update from remote peer bf60ee870f894a13beef385cf9a3d420: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:34:41.259776 19463 consensus_queue.cc:1035] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:34:41.259981 19469 consensus_queue.cc:1035] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:34:41.322065 18764 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:34:41.322289 18764 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:34:41.325194 18764 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3: Sending DeleteTablet for 3 replicas of tablet 868eb07e00384b17a587eba358f4bf30
I20260808 04:34:41.326097 18948 tablet_service.cc:1514] Processing DeleteTablet for tablet 868eb07e00384b17a587eba358f4bf30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:41 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:49538
I20260808 04:34:41.326385 19153 tablet_service.cc:1514] Processing DeleteTablet for tablet 868eb07e00384b17a587eba358f4bf30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:41 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:39728
I20260808 04:34:41.326882 19516 tablet_replica.cc:330] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420: stopping tablet replica
I20260808 04:34:41.327016 19331 tablet_service.cc:1514] Processing DeleteTablet for tablet 868eb07e00384b17a587eba358f4bf30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:41 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:46904
I20260808 04:34:41.327284 19516 raft_consensus.cc:2229] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:34:41.327696 19516 raft_consensus.cc:2258] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:34:41.328032 19519 tablet_replica.cc:330] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140: stopping tablet replica
I20260808 04:34:41.328316 19519 raft_consensus.cc:2229] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:34:41.328540 19519 raft_consensus.cc:2258] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:34:41.328450 18764 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 04:34:41.328910 19518 tablet_replica.cc:330] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1: stopping tablet replica
W20260808 04:34:41.329015 18764 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:34:41.329445 19518 raft_consensus.cc:2229] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:34:41.329939 19518 raft_consensus.cc:2258] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:34:41.331494 19519 ts_tablet_manager.cc:1905] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:34:41.332403 19518 ts_tablet_manager.cc:1905] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:34:41.333494 19516 ts_tablet_manager.cc:1905] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:34:41.335929 19153 tablet_service.cc:1467] Processing CreateTablet for tablet 264af9d5a1cb48d6b73a6e0f30eb7067 (DEFAULT_TABLE table=test-table [id=b5420bde416f4168b48546f975e0482c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:41.336354 19153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 264af9d5a1cb48d6b73a6e0f30eb7067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:41.336915 19331 tablet_service.cc:1467] Processing CreateTablet for tablet 264af9d5a1cb48d6b73a6e0f30eb7067 (DEFAULT_TABLE table=test-table [id=b5420bde416f4168b48546f975e0482c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:41.335929 18948 tablet_service.cc:1467] Processing CreateTablet for tablet 264af9d5a1cb48d6b73a6e0f30eb7067 (DEFAULT_TABLE table=test-table [id=b5420bde416f4168b48546f975e0482c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:41.337815 18948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 264af9d5a1cb48d6b73a6e0f30eb7067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:41.338106 19518 ts_tablet_manager.cc:1918] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:34:41.338234 19518 log.cc:1196] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/868eb07e00384b17a587eba358f4bf30
I20260808 04:34:41.338583 19518 ts_tablet_manager.cc:1939] T 868eb07e00384b17a587eba358f4bf30 P 17e39b4aa5ec48008e4477b502dda3b1: Deleting consensus metadata
I20260808 04:34:41.340420 18745 catalog_manager.cc:4841] TS 17e39b4aa5ec48008e4477b502dda3b1 (127.15.228.66:35549): tablet 868eb07e00384b17a587eba358f4bf30 (table test-table [id=6d2ad075549b4cf3bbdc1dc154fe3496]) successfully deleted
I20260808 04:34:41.341009 19457 tablet_bootstrap.cc:492] T 264af9d5a1cb48d6b73a6e0f30eb7067 P bf60ee870f894a13beef385cf9a3d420: Bootstrap starting.
I20260808 04:34:41.341028 19458 tablet_bootstrap.cc:492] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 17e39b4aa5ec48008e4477b502dda3b1: Bootstrap starting.
I20260808 04:34:41.342100 19458 tablet_bootstrap.cc:654] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 17e39b4aa5ec48008e4477b502dda3b1: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:41.342100 19457 tablet_bootstrap.cc:654] T 264af9d5a1cb48d6b73a6e0f30eb7067 P bf60ee870f894a13beef385cf9a3d420: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:41.342881 19331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 264af9d5a1cb48d6b73a6e0f30eb7067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:41.343839 19516 ts_tablet_manager.cc:1918] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:34:41.343938 19516 log.cc:1196] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-0/wal/wals/868eb07e00384b17a587eba358f4bf30
I20260808 04:34:41.344293 19516 ts_tablet_manager.cc:1939] T 868eb07e00384b17a587eba358f4bf30 P bf60ee870f894a13beef385cf9a3d420: Deleting consensus metadata
I20260808 04:34:41.344792 19457 tablet_bootstrap.cc:492] T 264af9d5a1cb48d6b73a6e0f30eb7067 P bf60ee870f894a13beef385cf9a3d420: No bootstrap required, opened a new log
I20260808 04:34:41.344888 19457 ts_tablet_manager.cc:1397] T 264af9d5a1cb48d6b73a6e0f30eb7067 P bf60ee870f894a13beef385cf9a3d420: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 04:34:41.345392 19457 raft_consensus.cc:348] T 264af9d5a1cb48d6b73a6e0f30eb7067 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.345525 19457 raft_consensus.cc:374] T 264af9d5a1cb48d6b73a6e0f30eb7067 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:41.345566 19457 raft_consensus.cc:729] T 264af9d5a1cb48d6b73a6e0f30eb7067 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf60ee870f894a13beef385cf9a3d420, State: Initialized, Role: FOLLOWER
I20260808 04:34:41.345633 18748 catalog_manager.cc:4841] TS bf60ee870f894a13beef385cf9a3d420 (127.15.228.65:38213): tablet 868eb07e00384b17a587eba358f4bf30 (table test-table [id=6d2ad075549b4cf3bbdc1dc154fe3496]) successfully deleted
I20260808 04:34:41.345726 19457 consensus_queue.cc:260] T 264af9d5a1cb48d6b73a6e0f30eb7067 P bf60ee870f894a13beef385cf9a3d420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.346503 19460 tablet_bootstrap.cc:492] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140: Bootstrap starting.
I20260808 04:34:41.347285 19458 tablet_bootstrap.cc:492] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 17e39b4aa5ec48008e4477b502dda3b1: No bootstrap required, opened a new log
I20260808 04:34:41.347385 19458 ts_tablet_manager.cc:1397] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 17e39b4aa5ec48008e4477b502dda3b1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260808 04:34:41.347970 19458 raft_consensus.cc:348] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.348078 19460 tablet_bootstrap.cc:654] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:41.348094 19458 raft_consensus.cc:374] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:41.348129 19458 raft_consensus.cc:729] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17e39b4aa5ec48008e4477b502dda3b1, State: Initialized, Role: FOLLOWER
I20260808 04:34:41.348261 19458 consensus_queue.cc:260] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 17e39b4aa5ec48008e4477b502dda3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.348515 19458 ts_tablet_manager.cc:1428] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 17e39b4aa5ec48008e4477b502dda3b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:34:41.348641 19519 ts_tablet_manager.cc:1918] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:34:41.348770 19519 log.cc:1196] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/868eb07e00384b17a587eba358f4bf30
I20260808 04:34:41.349432 19519 ts_tablet_manager.cc:1939] T 868eb07e00384b17a587eba358f4bf30 P 352419849eb14ccea05ab379f0d48140: Deleting consensus metadata
I20260808 04:34:41.352196 19457 ts_tablet_manager.cc:1428] T 264af9d5a1cb48d6b73a6e0f30eb7067 P bf60ee870f894a13beef385cf9a3d420: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260808 04:34:41.352744 18747 catalog_manager.cc:4841] TS 352419849eb14ccea05ab379f0d48140 (127.15.228.67:40013): tablet 868eb07e00384b17a587eba358f4bf30 (table test-table [id=6d2ad075549b4cf3bbdc1dc154fe3496]) successfully deleted
I20260808 04:34:41.353211 19460 tablet_bootstrap.cc:492] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140: No bootstrap required, opened a new log
I20260808 04:34:41.353353 19460 ts_tablet_manager.cc:1397] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20260808 04:34:41.354229 19460 raft_consensus.cc:348] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.354398 19460 raft_consensus.cc:374] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:41.354442 19460 raft_consensus.cc:729] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352419849eb14ccea05ab379f0d48140, State: Initialized, Role: FOLLOWER
I20260808 04:34:41.354614 19460 consensus_queue.cc:260] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.354996 19460 ts_tablet_manager.cc:1428] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 04:34:41.567411 19464 raft_consensus.cc:482] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:34:41.567653 19464 raft_consensus.cc:504] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.569489 19464 leader_election.cc:288] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf60ee870f894a13beef385cf9a3d420 (127.15.228.65:38213), 17e39b4aa5ec48008e4477b502dda3b1 (127.15.228.66:35549)
I20260808 04:34:41.574627 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:41.569905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:38213 (local address 127.15.228.67:37297)
0808 04:34:41.570533 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:41.570549 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:41.570730 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:41.571148 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:41.571151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:41.571168 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:41.571348 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:41.571355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.572262 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.572265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:41.573120 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:41.573130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.573876 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.573880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:41.574092 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:41.574147 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 04:34:41.574241 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 04:34:41.574351 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":359,"thread_start_us":70,"threads_started":1}
I20260808 04:34:41.574769 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:41.570328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:37297 (local address 127.15.228.65:38213)
0808 04:34:41.570486 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:41.570492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:41.570808 (+ 316us) server_negotiation.cc:407] Connection header received
0808 04:34:41.570899 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:41.570902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:41.570992 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:41.571112 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:41.571520 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.572094 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.573272 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.573703 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.573803 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:41.573852 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:41.574359 (+ 507us) server_negotiation.cc:299] Negotiation successful
0808 04:34:41.574555 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 04:34:41.575084 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:41.570062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:43687 (local address 127.15.228.66:35549)
0808 04:34:41.570447 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:41.570454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:41.570480 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:34:41.570700 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:41.570704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:41.570765 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:41.570876 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:41.571332 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.571967 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.573408 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.574067 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.574249 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:41.574317 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:41.574741 (+ 424us) server_negotiation.cc:299] Negotiation successful
0808 04:34:41.574854 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":119,"threads_started":1}
I20260808 04:34:41.575518 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:41.569894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:35549 (local address 127.15.228.67:43687)
0808 04:34:41.570330 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:41.570352 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:41.570523 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:41.570919 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:41.570922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:41.570980 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:41.571167 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:41.571176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.572132 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.572139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:41.573117 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:41.573127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.574355 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.574360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:41.574555 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:41.574585 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 04:34:41.574691 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 04:34:41.575370 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":40,"thread_start_us":112,"threads_started":1}
I20260808 04:34:41.575553 18983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "264af9d5a1cb48d6b73a6e0f30eb7067" candidate_uuid: "352419849eb14ccea05ab379f0d48140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf60ee870f894a13beef385cf9a3d420" is_pre_election: true
I20260808 04:34:41.575973 18983 raft_consensus.cc:2454] T 264af9d5a1cb48d6b73a6e0f30eb7067 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 352419849eb14ccea05ab379f0d48140 in term 0.
I20260808 04:34:41.576205 19183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "264af9d5a1cb48d6b73a6e0f30eb7067" candidate_uuid: "352419849eb14ccea05ab379f0d48140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17e39b4aa5ec48008e4477b502dda3b1" is_pre_election: true
I20260808 04:34:41.576450 19183 raft_consensus.cc:2454] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 352419849eb14ccea05ab379f0d48140 in term 0.
I20260808 04:34:41.576726 19277 leader_election.cc:302] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 352419849eb14ccea05ab379f0d48140, bf60ee870f894a13beef385cf9a3d420; no voters:
I20260808 04:34:41.577113 19464 raft_consensus.cc:2781] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:34:41.577200 19464 raft_consensus.cc:482] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:34:41.577258 19464 raft_consensus.cc:3037] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:41.578374 19464 raft_consensus.cc:504] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.578797 19464 leader_election.cc:288] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [CANDIDATE]: Term 1 election: Requested vote from peers bf60ee870f894a13beef385cf9a3d420 (127.15.228.65:38213), 17e39b4aa5ec48008e4477b502dda3b1 (127.15.228.66:35549)
I20260808 04:34:41.579280 19183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "264af9d5a1cb48d6b73a6e0f30eb7067" candidate_uuid: "352419849eb14ccea05ab379f0d48140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17e39b4aa5ec48008e4477b502dda3b1"
I20260808 04:34:41.579315 18983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "264af9d5a1cb48d6b73a6e0f30eb7067" candidate_uuid: "352419849eb14ccea05ab379f0d48140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf60ee870f894a13beef385cf9a3d420"
I20260808 04:34:41.579422 19183 raft_consensus.cc:3037] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:41.579486 18983 raft_consensus.cc:3037] T 264af9d5a1cb48d6b73a6e0f30eb7067 P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:41.581236 19183 raft_consensus.cc:2454] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 17e39b4aa5ec48008e4477b502dda3b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 352419849eb14ccea05ab379f0d48140 in term 1.
I20260808 04:34:41.581288 18983 raft_consensus.cc:2454] T 264af9d5a1cb48d6b73a6e0f30eb7067 P bf60ee870f894a13beef385cf9a3d420 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 352419849eb14ccea05ab379f0d48140 in term 1.
I20260808 04:34:41.581677 19274 leader_election.cc:302] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17e39b4aa5ec48008e4477b502dda3b1, 352419849eb14ccea05ab379f0d48140; no voters:
I20260808 04:34:41.581966 19464 raft_consensus.cc:2781] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:34:41.582238 19464 raft_consensus.cc:686] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [term 1 LEADER]: Becoming Leader. State: Replica: 352419849eb14ccea05ab379f0d48140, State: Running, Role: LEADER
I20260808 04:34:41.582708 19464 consensus_queue.cc:237] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:41.585664 18763 catalog_manager.cc:5495] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 reported cstate change: term changed from 0 to 1, leader changed from <none> to 352419849eb14ccea05ab379f0d48140 (127.15.228.67). New cstate: current_term: 1 leader_uuid: "352419849eb14ccea05ab379f0d48140" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } health_report { overall_health: HEALTHY } } }
I20260808 04:34:41.609506 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:41.603445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46910 (local address 127.15.228.67:40013)
0808 04:34:41.603926 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:41.603934 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:41.603946 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:34:41.604019 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:41.604021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:41.604076 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:41.604189 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:34:41.604622 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.605495 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.606773 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.607160 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.607264 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:41.607846 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:34:41.608695 (+ 849us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:34:41.608834 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:41.609079 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 04:34:41.609251 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":312,"threads_started":1}
I20260808 04:34:41.624455 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:41.620695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:60495 (local address 127.15.228.65:38213)
0808 04:34:41.620888 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:41.620895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:41.621152 (+ 257us) server_negotiation.cc:407] Connection header received
0808 04:34:41.621334 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:41.621337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:41.621398 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:41.621488 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:34:41.621756 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.622288 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.623264 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.623503 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.623617 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:41.623910 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:34:41.624015 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:34:41.624092 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:41.624255 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 04:34:41.624323 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:34:41.624609 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:41.620529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:38213 (local address 127.15.228.67:60495)
0808 04:34:41.621053 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:41.621067 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:41.621217 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:41.621502 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:41.621505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:41.621514 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:34:41.621649 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:41.621654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.622410 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.622414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:41.623146 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:41.623155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.623623 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.623627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:41.623719 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:41.623766 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:34:41.624130 (+ 364us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:34:41.624138 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 04:34:41.624225 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 04:34:41.624315 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":221,"threads_started":1}
I20260808 04:34:41.633862 18983 raft_consensus.cc:1261] T 264af9d5a1cb48d6b73a6e0f30eb7067 P bf60ee870f894a13beef385cf9a3d420 [term 1 FOLLOWER]: Refusing update from remote peer 352419849eb14ccea05ab379f0d48140: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:34:41.634560 19183 raft_consensus.cc:1261] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 17e39b4aa5ec48008e4477b502dda3b1 [term 1 FOLLOWER]: Refusing update from remote peer 352419849eb14ccea05ab379f0d48140: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:34:41.634984 19529 consensus_queue.cc:1035] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:34:41.635169 19464 consensus_queue.cc:1035] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:34:41.659893 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:41.656414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:40013 (local address 127.15.228.65:55475)
0808 04:34:41.656545 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:41.656561 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:41.656656 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:41.657025 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:41.657029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:41.657039 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:34:41.657139 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:41.657144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.658027 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.658032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:41.658637 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:41.658645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.659190 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.659194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:41.659279 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:41.659316 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:34:41.659614 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:34:41.659621 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 04:34:41.659682 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 04:34:41.659765 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260808 04:34:41.659884 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:41.656452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:55475 (local address 127.15.228.67:40013)
0808 04:34:41.656570 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:41.656574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:41.656611 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:34:41.656724 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:41.656726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:41.656765 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:41.656879 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:34:41.657248 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.657914 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.658761 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:41.659076 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:41.659191 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:41.659439 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:34:41.659521 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:34:41.659597 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:41.659693 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 04:34:41.659743 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260808 04:34:41.660985 19498 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 04:34:43.240769 18763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:34:43.240998 18763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:34:43.244083 18763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3: Sending DeleteTablet for 3 replicas of tablet 264af9d5a1cb48d6b73a6e0f30eb7067
I20260808 04:34:43.244756 18948 tablet_service.cc:1514] Processing DeleteTablet for tablet 264af9d5a1cb48d6b73a6e0f30eb7067 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:43 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:49538
I20260808 04:34:43.245080 19331 tablet_service.cc:1514] Processing DeleteTablet for tablet 264af9d5a1cb48d6b73a6e0f30eb7067 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:43 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:46904
I20260808 04:34:43.245122 19153 tablet_service.cc:1514] Processing DeleteTablet for tablet 264af9d5a1cb48d6b73a6e0f30eb7067 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:43 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:39728
I20260808 04:34:43.245389 19575 tablet_replica.cc:330] T 264af9d5a1cb48d6b73a6e0f30eb7067 P bf60ee870f894a13beef385cf9a3d420: stopping tablet replica
I20260808 04:34:43.245663 19575 raft_consensus.cc:2229] T 264af9d5a1cb48d6b73a6e0f30eb7067 P bf60ee870f894a13beef385cf9a3d420 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:34:43.245860 19575 raft_consensus.cc:2258] T 264af9d5a1cb48d6b73a6e0f30eb7067 P bf60ee870f894a13beef385cf9a3d420 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:34:43.246054 19576 tablet_replica.cc:330] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 17e39b4aa5ec48008e4477b502dda3b1: stopping tablet replica
I20260808 04:34:43.246208 19576 raft_consensus.cc:2229] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 17e39b4aa5ec48008e4477b502dda3b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:34:43.246327 19576 raft_consensus.cc:2258] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 17e39b4aa5ec48008e4477b502dda3b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:34:43.247148 19577 tablet_replica.cc:330] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140: stopping tablet replica
I20260808 04:34:43.247153 19576 ts_tablet_manager.cc:1905] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 17e39b4aa5ec48008e4477b502dda3b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:34:43.247323 19577 raft_consensus.cc:2229] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:34:43.247702 19577 raft_consensus.cc:2258] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:34:43.248572 19577 ts_tablet_manager.cc:1905] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:34:43.250808 19576 ts_tablet_manager.cc:1918] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 17e39b4aa5ec48008e4477b502dda3b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:34:43.250998 19576 log.cc:1196] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 17e39b4aa5ec48008e4477b502dda3b1: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/264af9d5a1cb48d6b73a6e0f30eb7067
I20260808 04:34:43.251466 19576 ts_tablet_manager.cc:1939] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 17e39b4aa5ec48008e4477b502dda3b1: Deleting consensus metadata
I20260808 04:34:43.252204 19577 ts_tablet_manager.cc:1918] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:34:43.252338 19577 log.cc:1196] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/264af9d5a1cb48d6b73a6e0f30eb7067
I20260808 04:34:43.252606 18745 catalog_manager.cc:4841] TS 17e39b4aa5ec48008e4477b502dda3b1 (127.15.228.66:35549): tablet 264af9d5a1cb48d6b73a6e0f30eb7067 (table test-table [id=b5420bde416f4168b48546f975e0482c]) successfully deleted
I20260808 04:34:43.252751 19577 ts_tablet_manager.cc:1939] T 264af9d5a1cb48d6b73a6e0f30eb7067 P 352419849eb14ccea05ab379f0d48140: Deleting consensus metadata
I20260808 04:34:43.253002 19575 ts_tablet_manager.cc:1905] T 264af9d5a1cb48d6b73a6e0f30eb7067 P bf60ee870f894a13beef385cf9a3d420: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:34:43.253875 18747 catalog_manager.cc:4841] TS 352419849eb14ccea05ab379f0d48140 (127.15.228.67:40013): tablet 264af9d5a1cb48d6b73a6e0f30eb7067 (table test-table [id=b5420bde416f4168b48546f975e0482c]) successfully deleted
I20260808 04:34:43.257517 19575 ts_tablet_manager.cc:1918] T 264af9d5a1cb48d6b73a6e0f30eb7067 P bf60ee870f894a13beef385cf9a3d420: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:34:43.257627 19575 log.cc:1196] T 264af9d5a1cb48d6b73a6e0f30eb7067 P bf60ee870f894a13beef385cf9a3d420: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-0/wal/wals/264af9d5a1cb48d6b73a6e0f30eb7067
I20260808 04:34:43.258020 19575 ts_tablet_manager.cc:1939] T 264af9d5a1cb48d6b73a6e0f30eb7067 P bf60ee870f894a13beef385cf9a3d420: Deleting consensus metadata
I20260808 04:34:43.258870 18748 catalog_manager.cc:4841] TS bf60ee870f894a13beef385cf9a3d420 (127.15.228.65:38213): tablet 264af9d5a1cb48d6b73a6e0f30eb7067 (table test-table [id=b5420bde416f4168b48546f975e0482c]) successfully deleted
Aug 08 04:34:43 dist-test-slave-drl0 krb5kdc[18710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163683, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 04:34:43 dist-test-slave-drl0 krb5kdc[18710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163683, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:43.283303 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:43.271721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.15.228.126:36681)
0808 04:34:43.272091 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:43.272100 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:43.272118 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:34:43.272214 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:43.272217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:43.272287 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:43.272401 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:43.273553 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:43.274163 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:43.275452 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:43.275688 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:43.275784 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:43.278590 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:43.278593 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:43.278596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:43.278625 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:43.280464 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:43.281207 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:43.281211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:43.281213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:43.281302 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:43.281911 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:43.281918 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:43.281920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:43.282329 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:43.282468 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:43.283034 (+ 566us) server_negotiation.cc:299] Negotiation successful
0808 04:34:43.283159 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":160,"threads_started":1}
I20260808 04:34:43.287593 18763 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:34:43.288586 18763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:34:43.296640 19153 tablet_service.cc:1467] Processing CreateTablet for tablet 3c326f1b6c9044a2bcefb0bf06b6973c (DEFAULT_TABLE table=test-table [id=e4f12b5131e64e4cb88ed45570102de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:43.296672 18948 tablet_service.cc:1467] Processing CreateTablet for tablet 3c326f1b6c9044a2bcefb0bf06b6973c (DEFAULT_TABLE table=test-table [id=e4f12b5131e64e4cb88ed45570102de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:43.296732 19331 tablet_service.cc:1467] Processing CreateTablet for tablet 3c326f1b6c9044a2bcefb0bf06b6973c (DEFAULT_TABLE table=test-table [id=e4f12b5131e64e4cb88ed45570102de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:43.297012 19331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c326f1b6c9044a2bcefb0bf06b6973c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:43.297015 18948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c326f1b6c9044a2bcefb0bf06b6973c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:43.297029 19153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c326f1b6c9044a2bcefb0bf06b6973c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:43.300004 19596 tablet_bootstrap.cc:492] T 3c326f1b6c9044a2bcefb0bf06b6973c P 17e39b4aa5ec48008e4477b502dda3b1: Bootstrap starting.
I20260808 04:34:43.300233 19595 tablet_bootstrap.cc:492] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420: Bootstrap starting.
I20260808 04:34:43.300556 19593 tablet_bootstrap.cc:492] T 3c326f1b6c9044a2bcefb0bf06b6973c P 352419849eb14ccea05ab379f0d48140: Bootstrap starting.
I20260808 04:34:43.301142 19595 tablet_bootstrap.cc:654] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:43.301474 19596 tablet_bootstrap.cc:654] T 3c326f1b6c9044a2bcefb0bf06b6973c P 17e39b4aa5ec48008e4477b502dda3b1: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:43.301497 19593 tablet_bootstrap.cc:654] T 3c326f1b6c9044a2bcefb0bf06b6973c P 352419849eb14ccea05ab379f0d48140: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:43.302603 19595 tablet_bootstrap.cc:492] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420: No bootstrap required, opened a new log
I20260808 04:34:43.302891 19593 tablet_bootstrap.cc:492] T 3c326f1b6c9044a2bcefb0bf06b6973c P 352419849eb14ccea05ab379f0d48140: No bootstrap required, opened a new log
I20260808 04:34:43.302913 19595 ts_tablet_manager.cc:1397] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:34:43.303063 19596 tablet_bootstrap.cc:492] T 3c326f1b6c9044a2bcefb0bf06b6973c P 17e39b4aa5ec48008e4477b502dda3b1: No bootstrap required, opened a new log
I20260808 04:34:43.303154 19596 ts_tablet_manager.cc:1397] T 3c326f1b6c9044a2bcefb0bf06b6973c P 17e39b4aa5ec48008e4477b502dda3b1: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260808 04:34:43.303155 19593 ts_tablet_manager.cc:1397] T 3c326f1b6c9044a2bcefb0bf06b6973c P 352419849eb14ccea05ab379f0d48140: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260808 04:34:43.303618 19596 raft_consensus.cc:348] T 3c326f1b6c9044a2bcefb0bf06b6973c P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:43.303618 19593 raft_consensus.cc:348] T 3c326f1b6c9044a2bcefb0bf06b6973c P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:43.303747 19596 raft_consensus.cc:374] T 3c326f1b6c9044a2bcefb0bf06b6973c P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:43.303747 19593 raft_consensus.cc:374] T 3c326f1b6c9044a2bcefb0bf06b6973c P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:43.303678 19595 raft_consensus.cc:348] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:43.303785 19596 raft_consensus.cc:729] T 3c326f1b6c9044a2bcefb0bf06b6973c P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17e39b4aa5ec48008e4477b502dda3b1, State: Initialized, Role: FOLLOWER
I20260808 04:34:43.303785 19593 raft_consensus.cc:729] T 3c326f1b6c9044a2bcefb0bf06b6973c P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352419849eb14ccea05ab379f0d48140, State: Initialized, Role: FOLLOWER
I20260808 04:34:43.303824 19595 raft_consensus.cc:374] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:43.303857 19595 raft_consensus.cc:729] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf60ee870f894a13beef385cf9a3d420, State: Initialized, Role: FOLLOWER
I20260808 04:34:43.303929 19596 consensus_queue.cc:260] T 3c326f1b6c9044a2bcefb0bf06b6973c P 17e39b4aa5ec48008e4477b502dda3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:43.303966 19593 consensus_queue.cc:260] T 3c326f1b6c9044a2bcefb0bf06b6973c P 352419849eb14ccea05ab379f0d48140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:43.304265 19593 ts_tablet_manager.cc:1428] T 3c326f1b6c9044a2bcefb0bf06b6973c P 352419849eb14ccea05ab379f0d48140: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 04:34:43.304325 19596 ts_tablet_manager.cc:1428] T 3c326f1b6c9044a2bcefb0bf06b6973c P 17e39b4aa5ec48008e4477b502dda3b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:34:43.304056 19595 consensus_queue.cc:260] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:43.305336 19595 ts_tablet_manager.cc:1428] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 04:34:43.561774 19553 raft_consensus.cc:482] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:34:43.561964 19553 raft_consensus.cc:504] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:43.562481 19553 leader_election.cc:288] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17e39b4aa5ec48008e4477b502dda3b1 (127.15.228.66:35549), 352419849eb14ccea05ab379f0d48140 (127.15.228.67:40013)
I20260808 04:34:43.562999 19183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c326f1b6c9044a2bcefb0bf06b6973c" candidate_uuid: "bf60ee870f894a13beef385cf9a3d420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17e39b4aa5ec48008e4477b502dda3b1" is_pre_election: true
I20260808 04:34:43.563189 19183 raft_consensus.cc:2454] T 3c326f1b6c9044a2bcefb0bf06b6973c P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf60ee870f894a13beef385cf9a3d420 in term 0.
I20260808 04:34:43.563131 19364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c326f1b6c9044a2bcefb0bf06b6973c" candidate_uuid: "bf60ee870f894a13beef385cf9a3d420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "352419849eb14ccea05ab379f0d48140" is_pre_election: true
I20260808 04:34:43.563316 19364 raft_consensus.cc:2454] T 3c326f1b6c9044a2bcefb0bf06b6973c P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf60ee870f894a13beef385cf9a3d420 in term 0.
I20260808 04:34:43.563573 18888 leader_election.cc:302] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17e39b4aa5ec48008e4477b502dda3b1, bf60ee870f894a13beef385cf9a3d420; no voters:
I20260808 04:34:43.563758 19553 raft_consensus.cc:2781] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:34:43.563823 19553 raft_consensus.cc:482] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:34:43.563864 19553 raft_consensus.cc:3037] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:43.565104 19553 raft_consensus.cc:504] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:43.565510 19553 leader_election.cc:288] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [CANDIDATE]: Term 1 election: Requested vote from peers 17e39b4aa5ec48008e4477b502dda3b1 (127.15.228.66:35549), 352419849eb14ccea05ab379f0d48140 (127.15.228.67:40013)
I20260808 04:34:43.565891 19183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c326f1b6c9044a2bcefb0bf06b6973c" candidate_uuid: "bf60ee870f894a13beef385cf9a3d420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17e39b4aa5ec48008e4477b502dda3b1"
I20260808 04:34:43.566037 19183 raft_consensus.cc:3037] T 3c326f1b6c9044a2bcefb0bf06b6973c P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:43.566082 19364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c326f1b6c9044a2bcefb0bf06b6973c" candidate_uuid: "bf60ee870f894a13beef385cf9a3d420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "352419849eb14ccea05ab379f0d48140"
I20260808 04:34:43.566213 19364 raft_consensus.cc:3037] T 3c326f1b6c9044a2bcefb0bf06b6973c P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:43.567104 19183 raft_consensus.cc:2454] T 3c326f1b6c9044a2bcefb0bf06b6973c P 17e39b4aa5ec48008e4477b502dda3b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf60ee870f894a13beef385cf9a3d420 in term 1.
I20260808 04:34:43.567405 19364 raft_consensus.cc:2454] T 3c326f1b6c9044a2bcefb0bf06b6973c P 352419849eb14ccea05ab379f0d48140 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf60ee870f894a13beef385cf9a3d420 in term 1.
I20260808 04:34:43.567410 18888 leader_election.cc:302] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17e39b4aa5ec48008e4477b502dda3b1, bf60ee870f894a13beef385cf9a3d420; no voters:
I20260808 04:34:43.567612 19553 raft_consensus.cc:2781] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:34:43.567955 19553 raft_consensus.cc:686] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [term 1 LEADER]: Becoming Leader. State: Replica: bf60ee870f894a13beef385cf9a3d420, State: Running, Role: LEADER
I20260808 04:34:43.568243 19553 consensus_queue.cc:237] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:43.571094 18766 catalog_manager.cc:5495] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf60ee870f894a13beef385cf9a3d420 (127.15.228.65). New cstate: current_term: 1 leader_uuid: "bf60ee870f894a13beef385cf9a3d420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } health_report { overall_health: UNKNOWN } } }
I20260808 04:34:43.616904 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:43.613190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53090 (local address 127.15.228.65:38213)
0808 04:34:43.613674 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:43.613682 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:43.613695 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:34:43.613734 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:43.613736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:43.613781 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:43.613887 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:34:43.614273 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:43.614845 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:43.615745 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:43.615987 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:43.616103 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:43.616384 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:34:43.616460 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:34:43.616545 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:43.616668 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 04:34:43.616726 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":265,"threads_started":1}
I20260808 04:34:43.624058 19364 raft_consensus.cc:1261] T 3c326f1b6c9044a2bcefb0bf06b6973c P 352419849eb14ccea05ab379f0d48140 [term 1 FOLLOWER]: Refusing update from remote peer bf60ee870f894a13beef385cf9a3d420: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:34:43.624105 19183 raft_consensus.cc:1261] T 3c326f1b6c9044a2bcefb0bf06b6973c P 17e39b4aa5ec48008e4477b502dda3b1 [term 1 FOLLOWER]: Refusing update from remote peer bf60ee870f894a13beef385cf9a3d420: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:34:43.624514 19469 consensus_queue.cc:1035] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:34:43.624744 19547 consensus_queue.cc:1035] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:34:43.662958 18763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:34:43.663123 18763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:34:43.665961 18763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3: Sending DeleteTablet for 3 replicas of tablet 3c326f1b6c9044a2bcefb0bf06b6973c
I20260808 04:34:43.667196 19331 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c326f1b6c9044a2bcefb0bf06b6973c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:43 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:46904
I20260808 04:34:43.667289 18948 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c326f1b6c9044a2bcefb0bf06b6973c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:43 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:49538
I20260808 04:34:43.667470 19575 tablet_replica.cc:330] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420: stopping tablet replica
I20260808 04:34:43.667505 19153 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c326f1b6c9044a2bcefb0bf06b6973c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:43 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:39728
I20260808 04:34:43.667603 19575 raft_consensus.cc:2229] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:34:43.667677 19576 tablet_replica.cc:330] T 3c326f1b6c9044a2bcefb0bf06b6973c P 17e39b4aa5ec48008e4477b502dda3b1: stopping tablet replica
I20260808 04:34:43.668483 19576 raft_consensus.cc:2229] T 3c326f1b6c9044a2bcefb0bf06b6973c P 17e39b4aa5ec48008e4477b502dda3b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:34:43.668525 19575 raft_consensus.cc:2258] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:34:43.668543 19577 tablet_replica.cc:330] T 3c326f1b6c9044a2bcefb0bf06b6973c P 352419849eb14ccea05ab379f0d48140: stopping tablet replica
I20260808 04:34:43.668639 19576 raft_consensus.cc:2258] T 3c326f1b6c9044a2bcefb0bf06b6973c P 17e39b4aa5ec48008e4477b502dda3b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:34:43.668771 19577 raft_consensus.cc:2229] T 3c326f1b6c9044a2bcefb0bf06b6973c P 352419849eb14ccea05ab379f0d48140 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:34:43.668956 19577 raft_consensus.cc:2258] T 3c326f1b6c9044a2bcefb0bf06b6973c P 352419849eb14ccea05ab379f0d48140 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:34:43.669504 19575 ts_tablet_manager.cc:1905] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:34:43.669548 18763 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 04:34:43.669924 19576 ts_tablet_manager.cc:1905] T 3c326f1b6c9044a2bcefb0bf06b6973c P 17e39b4aa5ec48008e4477b502dda3b1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 04:34:43.670172 18763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:34:43.671614 19577 ts_tablet_manager.cc:1905] T 3c326f1b6c9044a2bcefb0bf06b6973c P 352419849eb14ccea05ab379f0d48140: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:34:43.672676 19575 ts_tablet_manager.cc:1918] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:34:43.672768 19575 log.cc:1196] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-0/wal/wals/3c326f1b6c9044a2bcefb0bf06b6973c
I20260808 04:34:43.673095 19575 ts_tablet_manager.cc:1939] T 3c326f1b6c9044a2bcefb0bf06b6973c P bf60ee870f894a13beef385cf9a3d420: Deleting consensus metadata
I20260808 04:34:43.673364 19576 ts_tablet_manager.cc:1918] T 3c326f1b6c9044a2bcefb0bf06b6973c P 17e39b4aa5ec48008e4477b502dda3b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:34:43.673442 19576 log.cc:1196] T 3c326f1b6c9044a2bcefb0bf06b6973c P 17e39b4aa5ec48008e4477b502dda3b1: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/3c326f1b6c9044a2bcefb0bf06b6973c
I20260808 04:34:43.673703 19576 ts_tablet_manager.cc:1939] T 3c326f1b6c9044a2bcefb0bf06b6973c P 17e39b4aa5ec48008e4477b502dda3b1: Deleting consensus metadata
I20260808 04:34:43.673921 18748 catalog_manager.cc:4841] TS bf60ee870f894a13beef385cf9a3d420 (127.15.228.65:38213): tablet 3c326f1b6c9044a2bcefb0bf06b6973c (table test-table [id=e4f12b5131e64e4cb88ed45570102de5]) successfully deleted
I20260808 04:34:43.674440 18745 catalog_manager.cc:4841] TS 17e39b4aa5ec48008e4477b502dda3b1 (127.15.228.66:35549): tablet 3c326f1b6c9044a2bcefb0bf06b6973c (table test-table [id=e4f12b5131e64e4cb88ed45570102de5]) successfully deleted
I20260808 04:34:43.674718 19577 ts_tablet_manager.cc:1918] T 3c326f1b6c9044a2bcefb0bf06b6973c P 352419849eb14ccea05ab379f0d48140: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:34:43.674819 19577 log.cc:1196] T 3c326f1b6c9044a2bcefb0bf06b6973c P 352419849eb14ccea05ab379f0d48140: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/3c326f1b6c9044a2bcefb0bf06b6973c
I20260808 04:34:43.675151 19577 ts_tablet_manager.cc:1939] T 3c326f1b6c9044a2bcefb0bf06b6973c P 352419849eb14ccea05ab379f0d48140: Deleting consensus metadata
I20260808 04:34:43.675791 18948 tablet_service.cc:1467] Processing CreateTablet for tablet 03ad1704cfdf44cba9ec4135e2e6e80b (DEFAULT_TABLE table=test-table [id=4c13e56483f24fc19704fc91866f8542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:43.675980 18747 catalog_manager.cc:4841] TS 352419849eb14ccea05ab379f0d48140 (127.15.228.67:40013): tablet 3c326f1b6c9044a2bcefb0bf06b6973c (table test-table [id=e4f12b5131e64e4cb88ed45570102de5]) successfully deleted
I20260808 04:34:43.676082 18948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03ad1704cfdf44cba9ec4135e2e6e80b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:43.676288 19331 tablet_service.cc:1467] Processing CreateTablet for tablet 03ad1704cfdf44cba9ec4135e2e6e80b (DEFAULT_TABLE table=test-table [id=4c13e56483f24fc19704fc91866f8542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:43.676455 19153 tablet_service.cc:1467] Processing CreateTablet for tablet 03ad1704cfdf44cba9ec4135e2e6e80b (DEFAULT_TABLE table=test-table [id=4c13e56483f24fc19704fc91866f8542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:43.676538 19331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03ad1704cfdf44cba9ec4135e2e6e80b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:43.676679 19153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03ad1704cfdf44cba9ec4135e2e6e80b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:43.678220 19595 tablet_bootstrap.cc:492] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420: Bootstrap starting.
I20260808 04:34:43.678772 19593 tablet_bootstrap.cc:492] T 03ad1704cfdf44cba9ec4135e2e6e80b P 352419849eb14ccea05ab379f0d48140: Bootstrap starting.
I20260808 04:34:43.679137 19595 tablet_bootstrap.cc:654] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:43.679205 19596 tablet_bootstrap.cc:492] T 03ad1704cfdf44cba9ec4135e2e6e80b P 17e39b4aa5ec48008e4477b502dda3b1: Bootstrap starting.
I20260808 04:34:43.679950 19593 tablet_bootstrap.cc:654] T 03ad1704cfdf44cba9ec4135e2e6e80b P 352419849eb14ccea05ab379f0d48140: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:43.680545 19596 tablet_bootstrap.cc:654] T 03ad1704cfdf44cba9ec4135e2e6e80b P 17e39b4aa5ec48008e4477b502dda3b1: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:43.680876 19595 tablet_bootstrap.cc:492] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420: No bootstrap required, opened a new log
I20260808 04:34:43.681003 19595 ts_tablet_manager.cc:1397] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:34:43.681407 19595 raft_consensus.cc:348] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:43.681535 19595 raft_consensus.cc:374] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:43.681576 19595 raft_consensus.cc:729] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf60ee870f894a13beef385cf9a3d420, State: Initialized, Role: FOLLOWER
I20260808 04:34:43.681721 19595 consensus_queue.cc:260] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:43.681975 19596 tablet_bootstrap.cc:492] T 03ad1704cfdf44cba9ec4135e2e6e80b P 17e39b4aa5ec48008e4477b502dda3b1: No bootstrap required, opened a new log
I20260808 04:34:43.682025 19595 ts_tablet_manager.cc:1428] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 04:34:43.682073 19596 ts_tablet_manager.cc:1397] T 03ad1704cfdf44cba9ec4135e2e6e80b P 17e39b4aa5ec48008e4477b502dda3b1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:34:43.682570 19596 raft_consensus.cc:348] T 03ad1704cfdf44cba9ec4135e2e6e80b P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:43.682687 19596 raft_consensus.cc:374] T 03ad1704cfdf44cba9ec4135e2e6e80b P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:43.682732 19596 raft_consensus.cc:729] T 03ad1704cfdf44cba9ec4135e2e6e80b P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17e39b4aa5ec48008e4477b502dda3b1, State: Initialized, Role: FOLLOWER
I20260808 04:34:43.682864 19596 consensus_queue.cc:260] T 03ad1704cfdf44cba9ec4135e2e6e80b P 17e39b4aa5ec48008e4477b502dda3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:43.683125 19593 tablet_bootstrap.cc:492] T 03ad1704cfdf44cba9ec4135e2e6e80b P 352419849eb14ccea05ab379f0d48140: No bootstrap required, opened a new log
I20260808 04:34:43.683213 19593 ts_tablet_manager.cc:1397] T 03ad1704cfdf44cba9ec4135e2e6e80b P 352419849eb14ccea05ab379f0d48140: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260808 04:34:43.683288 19596 ts_tablet_manager.cc:1428] T 03ad1704cfdf44cba9ec4135e2e6e80b P 17e39b4aa5ec48008e4477b502dda3b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:34:43.683667 19593 raft_consensus.cc:348] T 03ad1704cfdf44cba9ec4135e2e6e80b P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:43.683807 19593 raft_consensus.cc:374] T 03ad1704cfdf44cba9ec4135e2e6e80b P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:43.683863 19593 raft_consensus.cc:729] T 03ad1704cfdf44cba9ec4135e2e6e80b P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352419849eb14ccea05ab379f0d48140, State: Initialized, Role: FOLLOWER
I20260808 04:34:43.683995 19593 consensus_queue.cc:260] T 03ad1704cfdf44cba9ec4135e2e6e80b P 352419849eb14ccea05ab379f0d48140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:43.684526 19593 ts_tablet_manager.cc:1428] T 03ad1704cfdf44cba9ec4135e2e6e80b P 352419849eb14ccea05ab379f0d48140: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:34:43.912060 19547 raft_consensus.cc:482] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:34:43.912251 19547 raft_consensus.cc:504] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:43.912849 19547 leader_election.cc:288] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17e39b4aa5ec48008e4477b502dda3b1 (127.15.228.66:35549), 352419849eb14ccea05ab379f0d48140 (127.15.228.67:40013)
I20260808 04:34:43.913440 19364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03ad1704cfdf44cba9ec4135e2e6e80b" candidate_uuid: "bf60ee870f894a13beef385cf9a3d420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "352419849eb14ccea05ab379f0d48140" is_pre_election: true
I20260808 04:34:43.913604 19364 raft_consensus.cc:2454] T 03ad1704cfdf44cba9ec4135e2e6e80b P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf60ee870f894a13beef385cf9a3d420 in term 0.
I20260808 04:34:43.913473 19183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03ad1704cfdf44cba9ec4135e2e6e80b" candidate_uuid: "bf60ee870f894a13beef385cf9a3d420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17e39b4aa5ec48008e4477b502dda3b1" is_pre_election: true
I20260808 04:34:43.913782 19183 raft_consensus.cc:2454] T 03ad1704cfdf44cba9ec4135e2e6e80b P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf60ee870f894a13beef385cf9a3d420 in term 0.
I20260808 04:34:43.913966 18890 leader_election.cc:302] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 352419849eb14ccea05ab379f0d48140, bf60ee870f894a13beef385cf9a3d420; no voters:
I20260808 04:34:43.914240 19547 raft_consensus.cc:2781] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:34:43.914317 19547 raft_consensus.cc:482] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:34:43.914367 19547 raft_consensus.cc:3037] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:43.916095 19547 raft_consensus.cc:504] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:43.916528 19547 leader_election.cc:288] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420 [CANDIDATE]: Term 1 election: Requested vote from peers 17e39b4aa5ec48008e4477b502dda3b1 (127.15.228.66:35549), 352419849eb14ccea05ab379f0d48140 (127.15.228.67:40013)
I20260808 04:34:43.916903 19183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03ad1704cfdf44cba9ec4135e2e6e80b" candidate_uuid: "bf60ee870f894a13beef385cf9a3d420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17e39b4aa5ec48008e4477b502dda3b1"
I20260808 04:34:43.917037 19183 raft_consensus.cc:3037] T 03ad1704cfdf44cba9ec4135e2e6e80b P 17e39b4aa5ec48008e4477b502dda3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:43.917044 19364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03ad1704cfdf44cba9ec4135e2e6e80b" candidate_uuid: "bf60ee870f894a13beef385cf9a3d420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "352419849eb14ccea05ab379f0d48140"
I20260808 04:34:43.917176 19364 raft_consensus.cc:3037] T 03ad1704cfdf44cba9ec4135e2e6e80b P 352419849eb14ccea05ab379f0d48140 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:43.918109 19364 raft_consensus.cc:2454] T 03ad1704cfdf44cba9ec4135e2e6e80b P 352419849eb14ccea05ab379f0d48140 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf60ee870f894a13beef385cf9a3d420 in term 1.
I20260808 04:34:43.918171 19183 raft_consensus.cc:2454] T 03ad1704cfdf44cba9ec4135e2e6e80b P 17e39b4aa5ec48008e4477b502dda3b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf60ee870f894a13beef385cf9a3d420 in term 1.
I20260808 04:34:43.918428 18890 leader_election.cc:302] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17e39b4aa5ec48008e4477b502dda3b1, 352419849eb14ccea05ab379f0d48140, bf60ee870f894a13beef385cf9a3d420; no voters:
I20260808 04:34:43.918658 19547 raft_consensus.cc:2781] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:34:43.918788 19547 raft_consensus.cc:686] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420 [term 1 LEADER]: Becoming Leader. State: Replica: bf60ee870f894a13beef385cf9a3d420, State: Running, Role: LEADER
I20260808 04:34:43.918985 19547 consensus_queue.cc:237] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } }
I20260808 04:34:43.920919 18764 catalog_manager.cc:5495] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf60ee870f894a13beef385cf9a3d420 (127.15.228.65). New cstate: current_term: 1 leader_uuid: "bf60ee870f894a13beef385cf9a3d420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf60ee870f894a13beef385cf9a3d420" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 } health_report { overall_health: UNKNOWN } } }
I20260808 04:34:43.943776 19364 raft_consensus.cc:1261] T 03ad1704cfdf44cba9ec4135e2e6e80b P 352419849eb14ccea05ab379f0d48140 [term 1 FOLLOWER]: Refusing update from remote peer bf60ee870f894a13beef385cf9a3d420: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:34:43.944217 19547 consensus_queue.cc:1035] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "352419849eb14ccea05ab379f0d48140" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:34:43.944376 19182 raft_consensus.cc:1261] T 03ad1704cfdf44cba9ec4135e2e6e80b P 17e39b4aa5ec48008e4477b502dda3b1 [term 1 FOLLOWER]: Refusing update from remote peer bf60ee870f894a13beef385cf9a3d420: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:34:43.944983 19547 consensus_queue.cc:1035] T 03ad1704cfdf44cba9ec4135e2e6e80b P bf60ee870f894a13beef385cf9a3d420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17e39b4aa5ec48008e4477b502dda3b1" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 35549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 04:34:43.961018 19545 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 04:34:45.527290 18764 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:34:45.527702 18764 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:34:45.531710 18764 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 111dbf0a4f274bb390450a906a6879c3: Sending DeleteTablet for 3 replicas of tablet 03ad1704cfdf44cba9ec4135e2e6e80b
I20260808 04:34:45.532668 19331 tablet_service.cc:1514] Processing DeleteTablet for tablet 03ad1704cfdf44cba9ec4135e2e6e80b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:45 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:46904
I20260808 04:34:45.532725 18948 tablet_service.cc:1514] Processing DeleteTablet for tablet 03ad1704cfdf44cba9ec4135e2e6e80b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:45 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:49538
I20260808 04:34:45.533336 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 18870
I20260808 04:34:45.533315 19153 tablet_service.cc:1514] Processing DeleteTablet for tablet 03ad1704cfdf44cba9ec4135e2e6e80b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:45 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:39728
I20260808 04:34:45.540711 19646 tablet_replica.cc:330] T 03ad1704cfdf44cba9ec4135e2e6e80b P 352419849eb14ccea05ab379f0d48140: stopping tablet replica
I20260808 04:34:45.540985 19646 raft_consensus.cc:2229] T 03ad1704cfdf44cba9ec4135e2e6e80b P 352419849eb14ccea05ab379f0d48140 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:34:45.541088 19648 tablet_replica.cc:330] T 03ad1704cfdf44cba9ec4135e2e6e80b P 17e39b4aa5ec48008e4477b502dda3b1: stopping tablet replica
I20260808 04:34:45.541173 19646 raft_consensus.cc:2258] T 03ad1704cfdf44cba9ec4135e2e6e80b P 352419849eb14ccea05ab379f0d48140 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:34:45.541260 19648 raft_consensus.cc:2229] T 03ad1704cfdf44cba9ec4135e2e6e80b P 17e39b4aa5ec48008e4477b502dda3b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:34:45.541453 19648 raft_consensus.cc:2258] T 03ad1704cfdf44cba9ec4135e2e6e80b P 17e39b4aa5ec48008e4477b502dda3b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:34:45.542034 19646 ts_tablet_manager.cc:1905] T 03ad1704cfdf44cba9ec4135e2e6e80b P 352419849eb14ccea05ab379f0d48140: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:34:45.542173 19648 ts_tablet_manager.cc:1905] T 03ad1704cfdf44cba9ec4135e2e6e80b P 17e39b4aa5ec48008e4477b502dda3b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:34:45.544471 19648 ts_tablet_manager.cc:1918] T 03ad1704cfdf44cba9ec4135e2e6e80b P 17e39b4aa5ec48008e4477b502dda3b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:34:45.544571 19648 log.cc:1196] T 03ad1704cfdf44cba9ec4135e2e6e80b P 17e39b4aa5ec48008e4477b502dda3b1: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/03ad1704cfdf44cba9ec4135e2e6e80b
I20260808 04:34:45.544899 19648 ts_tablet_manager.cc:1939] T 03ad1704cfdf44cba9ec4135e2e6e80b P 17e39b4aa5ec48008e4477b502dda3b1: Deleting consensus metadata
I20260808 04:34:45.545707 19646 ts_tablet_manager.cc:1918] T 03ad1704cfdf44cba9ec4135e2e6e80b P 352419849eb14ccea05ab379f0d48140: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 04:34:45.545804 19646 log.cc:1196] T 03ad1704cfdf44cba9ec4135e2e6e80b P 352419849eb14ccea05ab379f0d48140: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/03ad1704cfdf44cba9ec4135e2e6e80b
I20260808 04:34:45.545743 18745 catalog_manager.cc:4841] TS 17e39b4aa5ec48008e4477b502dda3b1 (127.15.228.66:35549): tablet 03ad1704cfdf44cba9ec4135e2e6e80b (table test-table [id=4c13e56483f24fc19704fc91866f8542]) successfully deleted
I20260808 04:34:45.546159 19646 ts_tablet_manager.cc:1939] T 03ad1704cfdf44cba9ec4135e2e6e80b P 352419849eb14ccea05ab379f0d48140: Deleting consensus metadata
I20260808 04:34:45.547056 18747 catalog_manager.cc:4841] TS 352419849eb14ccea05ab379f0d48140 (127.15.228.67:40013): tablet 03ad1704cfdf44cba9ec4135e2e6e80b (table test-table [id=4c13e56483f24fc19704fc91866f8542]) successfully deleted
W20260808 04:34:45.550565 18748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.228.65:38213 (error 108)
I20260808 04:34:45.551504 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 19065
I20260808 04:34:45.556373 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:45.551727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:38213 (local address 127.0.0.1:53096)
0808 04:34:45.556127 (+ 4400us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:45.556228 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.65:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4281,"thread_start_us":127,"threads_started":1}
W20260808 04:34:45.556684 18748 catalog_manager.cc:4584] TS bf60ee870f894a13beef385cf9a3d420 (127.15.228.65:38213): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 03ad1704cfdf44cba9ec4135e2e6e80b: Network error: Client connection negotiation failed: client connection to 127.15.228.65:38213: connect: Connection refused (error 111)
I20260808 04:34:45.573415 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 19249
I20260808 04:34:45.590261 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 18725
2026-08-08T04:34:45Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7337 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:34:45 dist-test-slave-drl0 krb5kdc[19657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:34:45 dist-test-slave-drl0 krb5kdc[19657](info): set up 2 sockets
Aug 08 04:34:45 dist-test-slave-drl0 krb5kdc[19657](info): commencing operation
krb5kdc: starting...
W20260808 04:34:46.949507 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.320s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260808 04:34:47.204192 16273 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.255s user 0.000s sys 0.003s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260808 04:34:47.517705 16273 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.227s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/test-user.keytab.
W20260808 04:34:48.089265 16273 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.571s user 0.000s sys 0.002s
Aug 08 04:34:48 dist-test-slave-drl0 krb5kdc[19657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:34:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:34:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:34:48.148490 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:33249
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:40603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:33249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:48.282372 19674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:48.282742 19674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:48.282825 19674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:48.287326 19674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:34:48.287488 19674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:48.287533 19674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:48.287593 19674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:34:48.287631 19674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:34:48.293709 19674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:40603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:33249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:33249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:48.295562 19674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:48.297022 19674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:48.308280 19684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:48.308463 19674 server_base.cc:1034] running on GCE node
W20260808 04:34:48.308333 19688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:48.308367 19686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:48.309284 19674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:48.310642 19674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:48.312085 19674 hybrid_clock.cc:648] HybridClock initialized: now 1786163688312054 us; error 65 us; skew 500 ppm
Aug 08 04:34:48 dist-test-slave-drl0 krb5kdc[19657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163688, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:48.315872 19674 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:34:48.316910 19674 webserver.cc:460] Webserver started at http://127.15.228.126:37801/ using document root <none> and password file <none>
I20260808 04:34:48.317589 19674 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:48.317662 19674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:48.317845 19674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:48.320171 19674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "a63a6c7b3d384959a4c3408720d3f40f"
format_stamp: "Formatted at 2026-08-08 04:34:48 on dist-test-slave-drl0"
server_key: "753f844e73459ab42d1d721d8a116157"
server_key_iv: "97fede916e5d7e3cf05581c88b4486ca"
server_key_version: "encryptionkey@0"
I20260808 04:34:48.320719 19674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "a63a6c7b3d384959a4c3408720d3f40f"
format_stamp: "Formatted at 2026-08-08 04:34:48 on dist-test-slave-drl0"
server_key: "753f844e73459ab42d1d721d8a116157"
server_key_iv: "97fede916e5d7e3cf05581c88b4486ca"
server_key_version: "encryptionkey@0"
I20260808 04:34:48.324699 19674 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 04:34:48.327917 19697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:48.329576 19674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:34:48.329708 19674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "a63a6c7b3d384959a4c3408720d3f40f"
format_stamp: "Formatted at 2026-08-08 04:34:48 on dist-test-slave-drl0"
server_key: "753f844e73459ab42d1d721d8a116157"
server_key_iv: "97fede916e5d7e3cf05581c88b4486ca"
server_key_version: "encryptionkey@0"
I20260808 04:34:48.329873 19674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:48.351351 19674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:48.355515 19674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:48.355787 19674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:48.364137 19674 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:33249
I20260808 04:34:48.364184 19768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:33249 every 8 connection(s)
I20260808 04:34:48.365170 19674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:34:48.367655 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 19674
I20260808 04:34:48.367857 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:34:48.368217 16273 external_mini_cluster.cc:1428] Setting key 5f15ae64596fb09e07375837a03b4b7d
I20260808 04:34:48.369940 19769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:48.379029 19769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f: Bootstrap starting.
Aug 08 04:34:48 dist-test-slave-drl0 krb5kdc[19657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:48.381758 19769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:48.382772 19769 log.cc:824] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:48.385363 19769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f: No bootstrap required, opened a new log
I20260808 04:34:48.386664 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:48.369953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59782 (local address 127.15.228.126:33249)
0808 04:34:48.370526 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:48.370555 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:48.370641 (+ 86us) server_negotiation.cc:407] Connection header received
0808 04:34:48.371716 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:48.371742 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:48.372267 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:48.372744 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:48.373819 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:48.375490 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:48.376723 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:48.377547 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:48.377726 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:48.380758 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:48.380765 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:48.380781 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:48.380814 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:48.383113 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:48.383805 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:48.383812 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:48.383818 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:48.383961 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:48.384323 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:48.384326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:48.384328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:48.384842 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:48.385027 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:48.385354 (+ 327us) server_negotiation.cc:299] Negotiation successful
0808 04:34:48.385663 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":179,"threads_started":1}
I20260808 04:34:48.389389 19769 raft_consensus.cc:348] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63a6c7b3d384959a4c3408720d3f40f" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33249 } }
I20260808 04:34:48.389667 19769 raft_consensus.cc:374] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:48.389708 19769 raft_consensus.cc:729] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a63a6c7b3d384959a4c3408720d3f40f, State: Initialized, Role: FOLLOWER
I20260808 04:34:48.390131 19769 consensus_queue.cc:260] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63a6c7b3d384959a4c3408720d3f40f" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33249 } }
I20260808 04:34:48.390271 19769 raft_consensus.cc:388] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:34:48.390313 19769 raft_consensus.cc:482] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:34:48.390388 19769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:48.391359 19769 raft_consensus.cc:504] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63a6c7b3d384959a4c3408720d3f40f" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33249 } }
I20260808 04:34:48.391654 19769 leader_election.cc:302] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a63a6c7b3d384959a4c3408720d3f40f; no voters:
I20260808 04:34:48.392012 19769 leader_election.cc:288] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:34:48.392385 19777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:34:48.392841 19777 raft_consensus.cc:686] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f [term 1 LEADER]: Becoming Leader. State: Replica: a63a6c7b3d384959a4c3408720d3f40f, State: Running, Role: LEADER
I20260808 04:34:48.393283 19777 consensus_queue.cc:237] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63a6c7b3d384959a4c3408720d3f40f" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33249 } }
I20260808 04:34:48.393399 19769 sys_catalog.cc:564] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:34:48.395303 19779 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a63a6c7b3d384959a4c3408720d3f40f. Latest consensus state: current_term: 1 leader_uuid: "a63a6c7b3d384959a4c3408720d3f40f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63a6c7b3d384959a4c3408720d3f40f" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33249 } } }
I20260808 04:34:48.395450 19779 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f [sys.catalog]: This master's current role is: LEADER
I20260808 04:34:48.395874 19778 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a63a6c7b3d384959a4c3408720d3f40f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63a6c7b3d384959a4c3408720d3f40f" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33249 } } }
I20260808 04:34:48.396301 19778 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f [sys.catalog]: This master's current role is: LEADER
W20260808 04:34:48.398525 19801 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:34:48.398591 19801 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:34:48.398677 19799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:34:48.399503 19799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:34:48.406230 19799 catalog_manager.cc:1347] Generated new cluster ID: 957ad584016b4aa3b72be40ebf2c6cb4
I20260808 04:34:48.406381 19799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:34:48.443722 19799 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:34:48.445446 19799 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:34:48.457566 19799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a63a6c7b3d384959a4c3408720d3f40f: Generated new TSK 0
I20260808 04:34:48.458372 19799 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
I20260808 04:34:48.506724 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33249
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:40603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:48.646747 19810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:48.647050 19810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:48.647097 19810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:48.651314 19810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:48.651388 19810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:48.651451 19810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:34:48.656899 19810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:40603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:48.658151 19810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:48.659296 19810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:48.670305 19818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:48.670354 19822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:48.670354 19819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:48.670439 19810 server_base.cc:1034] running on GCE node
I20260808 04:34:48.671326 19810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:48.672214 19810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:48.673521 19810 hybrid_clock.cc:648] HybridClock initialized: now 1786163688673434 us; error 105 us; skew 500 ppm
Aug 08 04:34:48 dist-test-slave-drl0 krb5kdc[19657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163688, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:48.677964 19810 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:34:48.679132 19810 webserver.cc:460] Webserver started at http://127.15.228.65:34223/ using document root <none> and password file <none>
I20260808 04:34:48.679791 19810 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:48.679863 19810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:48.680105 19810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:48.682647 19810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "cb1db2ada5514d93a7dcf5188f18c8a2"
format_stamp: "Formatted at 2026-08-08 04:34:48 on dist-test-slave-drl0"
server_key: "9210f595229468a7b0c39c99f9c62b9a"
server_key_iv: "70ecca80360c680a15f010ef91405072"
server_key_version: "encryptionkey@0"
I20260808 04:34:48.683323 19810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "cb1db2ada5514d93a7dcf5188f18c8a2"
format_stamp: "Formatted at 2026-08-08 04:34:48 on dist-test-slave-drl0"
server_key: "9210f595229468a7b0c39c99f9c62b9a"
server_key_iv: "70ecca80360c680a15f010ef91405072"
server_key_version: "encryptionkey@0"
I20260808 04:34:48.687451 19810 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:34:48.690642 19829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:48.692679 19810 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:34:48.693020 19810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "cb1db2ada5514d93a7dcf5188f18c8a2"
format_stamp: "Formatted at 2026-08-08 04:34:48 on dist-test-slave-drl0"
server_key: "9210f595229468a7b0c39c99f9c62b9a"
server_key_iv: "70ecca80360c680a15f010ef91405072"
server_key_version: "encryptionkey@0"
I20260808 04:34:48.693212 19810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:48.708559 19810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:48.712230 19810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:48.712461 19810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:48.713346 19810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:34:48.714532 19810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:48.714596 19810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:48.714643 19810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:48.714694 19810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:48.725111 19810 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:36023
I20260808 04:34:48.725133 19986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:36023 every 8 connection(s)
I20260808 04:34:48.726035 19810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:34:48.727205 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 19810
I20260808 04:34:48.727360 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:34:48.727797 16273 external_mini_cluster.cc:1428] Setting key b83adfbf08be428d9ae9b6b3d3ec01b0
Aug 08 04:34:48 dist-test-slave-drl0 krb5kdc[19657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163688, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.228.66@KRBTEST.COM; defaulting to no policy
I20260808 04:34:48.742590 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:48.729659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:36005 (local address 127.15.228.126:33249)
0808 04:34:48.729830 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:48.729836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:48.730200 (+ 364us) server_negotiation.cc:407] Connection header received
0808 04:34:48.731301 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:48.731305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:48.731369 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:48.731475 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:48.733022 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:48.733663 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:48.735148 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:48.735442 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:48.735594 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:48.738479 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:48.738483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:48.738485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:48.738509 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:48.740275 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:48.740917 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:48.740921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:48.740923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:48.741021 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:48.741378 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:48.741383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:48.741385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:48.741573 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:48.741687 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:48.742276 (+ 589us) server_negotiation.cc:299] Negotiation successful
0808 04:34:48.742412 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.66@KRBTEST.COM" created.
I20260808 04:34:48.743566 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:48.728688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:33249 (local address 127.15.228.65:36005)
0808 04:34:48.729883 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:48.729965 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:48.731079 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:48.731654 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:48.731661 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:48.732068 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:48.732819 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:48.732841 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:48.733814 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:48.733817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:48.734876 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:48.734886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:48.735643 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:48.735648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:48.735847 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:48.736541 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:48.736560 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:48.738285 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:48.740403 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:48.740409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:48.740425 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:48.740715 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:48.741123 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:48.741126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:48.741128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:48.741276 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:48.741687 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:48.741696 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:48.741989 (+ 293us) client_negotiation.cc:769] Sending connection context
0808 04:34:48.742223 (+ 234us) client_negotiation.cc:240] Negotiation successful
0808 04:34:48.742621 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":162,"threads_started":1}
I20260808 04:34:48.745040 19987 heartbeater.cc:347] Connected to a master server at 127.15.228.126:33249
I20260808 04:34:48.745327 19987 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:48.745860 19987 heartbeater.cc:511] Master 127.15.228.126:33249 requested a full tablet report, sending...
I20260808 04:34:48.748304 19722 ts_manager.cc:194] Registered new tserver with Master: cb1db2ada5514d93a7dcf5188f18c8a2 (127.15.228.65:36023)
I20260808 04:34:48.750191 19722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:36005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
I20260808 04:34:48.767060 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33249
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:40603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:48.902748 19994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:48.903105 19994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:48.903158 19994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:48.908538 19994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:48.908610 19994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:48.908674 19994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:34:48.914438 19994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:40603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:48.915900 19994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:48.917295 19994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:48.929268 20003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:48.929524 19994 server_base.cc:1034] running on GCE node
W20260808 04:34:48.929706 20006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:48.930476 20004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:48.931116 19994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:48.932016 19994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:48.933189 19994 hybrid_clock.cc:648] HybridClock initialized: now 1786163688933174 us; error 37 us; skew 500 ppm
Aug 08 04:34:48 dist-test-slave-drl0 krb5kdc[19657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163688, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:48.937383 19994 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:34:48.941021 19994 webserver.cc:460] Webserver started at http://127.15.228.66:40109/ using document root <none> and password file <none>
I20260808 04:34:48.942201 19994 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:48.942345 19994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:48.942653 19994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:48.946094 19994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "5d86c9fdce274a6f82a3c888292439d6"
format_stamp: "Formatted at 2026-08-08 04:34:48 on dist-test-slave-drl0"
server_key: "fed20895db62485eecc2fe7b53d5d944"
server_key_iv: "5274f0fa1a7b875417b56d7bcd40d67a"
server_key_version: "encryptionkey@0"
I20260808 04:34:48.946898 19994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "5d86c9fdce274a6f82a3c888292439d6"
format_stamp: "Formatted at 2026-08-08 04:34:48 on dist-test-slave-drl0"
server_key: "fed20895db62485eecc2fe7b53d5d944"
server_key_iv: "5274f0fa1a7b875417b56d7bcd40d67a"
server_key_version: "encryptionkey@0"
I20260808 04:34:48.953153 19994 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 04:34:48.957679 20013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:48.959906 19994 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:34:48.960103 19994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "5d86c9fdce274a6f82a3c888292439d6"
format_stamp: "Formatted at 2026-08-08 04:34:48 on dist-test-slave-drl0"
server_key: "fed20895db62485eecc2fe7b53d5d944"
server_key_iv: "5274f0fa1a7b875417b56d7bcd40d67a"
server_key_version: "encryptionkey@0"
I20260808 04:34:48.960269 19994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:48.984314 19994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:48.988235 19994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:48.988423 19994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:48.989037 19994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:34:48.989914 19994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:48.989966 19994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:48.990011 19994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:48.990042 19994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:49.004406 19994 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:46091
I20260808 04:34:49.004446 20163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:46091 every 8 connection(s)
I20260808 04:34:49.005445 19994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:34:49.008123 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 19994
I20260808 04:34:49.008303 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:34:49.008661 16273 external_mini_cluster.cc:1428] Setting key d4f822bff1486274c6e8d45179fff36e
Aug 08 04:34:49 dist-test-slave-drl0 krb5kdc[19657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163688, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.228.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.67@KRBTEST.COM" created.
I20260808 04:34:49.025224 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:49.008105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:45159 (local address 127.15.228.126:33249)
0808 04:34:49.008355 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:49.008364 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:49.009834 (+ 1470us) server_negotiation.cc:407] Connection header received
0808 04:34:49.011138 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:49.011141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:49.011210 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:49.011327 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:49.012978 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:49.013746 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:49.015875 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:49.016182 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:49.016330 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:49.020208 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:49.020212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:49.020215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:49.020250 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:49.022257 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:49.023079 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:49.023084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:49.023087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:49.023192 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:49.023612 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:49.023615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:49.023617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:49.023940 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:49.024072 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:49.024837 (+ 765us) server_negotiation.cc:299] Negotiation successful
0808 04:34:49.025012 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 04:34:49.026543 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:49.008472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:33249 (local address 127.15.228.66:45159)
0808 04:34:49.009533 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:49.009600 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:49.010801 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:49.011577 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:49.011586 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:49.012080 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:49.012780 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:49.012797 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:49.013944 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:49.013951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:49.015677 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:49.015697 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:49.016404 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:49.016410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:49.016643 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:49.017465 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:49.017500 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:49.019918 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:49.022437 (+ 2519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:49.022447 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:49.022480 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:49.022888 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:49.023300 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:49.023303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:49.023305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:49.023491 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:49.024078 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:49.024088 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:49.024498 (+ 410us) client_negotiation.cc:769] Sending connection context
0808 04:34:49.024779 (+ 281us) client_negotiation.cc:240] Negotiation successful
0808 04:34:49.025252 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":171,"threads_started":1}
I20260808 04:34:49.028338 20164 heartbeater.cc:347] Connected to a master server at 127.15.228.126:33249
I20260808 04:34:49.028697 20164 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:49.029626 20164 heartbeater.cc:511] Master 127.15.228.126:33249 requested a full tablet report, sending...
I20260808 04:34:49.031270 19722 ts_manager.cc:194] Registered new tserver with Master: 5d86c9fdce274a6f82a3c888292439d6 (127.15.228.66:46091)
I20260808 04:34:49.031981 19722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:45159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
I20260808 04:34:49.047173 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33249
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:40603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:49.182420 20172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:49.182792 20172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:49.182881 20172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:49.187186 20172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:49.187284 20172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:49.187389 20172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:34:49.194705 20172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:40603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:49.196084 20172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:49.197248 20172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:49.208730 20188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:49.208961 20184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:49.208750 20186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:49.209378 20172 server_base.cc:1034] running on GCE node
I20260808 04:34:49.209843 20172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:49.210536 20172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:49.211761 20172 hybrid_clock.cc:648] HybridClock initialized: now 1786163689211743 us; error 31 us; skew 500 ppm
Aug 08 04:34:49 dist-test-slave-drl0 krb5kdc[19657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163689, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:49.214851 20172 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:34:49.215943 20172 webserver.cc:460] Webserver started at http://127.15.228.67:42305/ using document root <none> and password file <none>
I20260808 04:34:49.216553 20172 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:49.216616 20172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:49.216837 20172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:49.219542 20172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "4866071b23cf4b3eb5b975490c404f5c"
format_stamp: "Formatted at 2026-08-08 04:34:49 on dist-test-slave-drl0"
server_key: "89860330234c288ddeb4b1ce17d28dc5"
server_key_iv: "78f942581590fbbee95cd146859a1393"
server_key_version: "encryptionkey@0"
I20260808 04:34:49.220165 20172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "4866071b23cf4b3eb5b975490c404f5c"
format_stamp: "Formatted at 2026-08-08 04:34:49 on dist-test-slave-drl0"
server_key: "89860330234c288ddeb4b1ce17d28dc5"
server_key_iv: "78f942581590fbbee95cd146859a1393"
server_key_version: "encryptionkey@0"
I20260808 04:34:49.224584 20172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:34:49.227492 20199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:49.229421 20172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:34:49.229776 20172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "4866071b23cf4b3eb5b975490c404f5c"
format_stamp: "Formatted at 2026-08-08 04:34:49 on dist-test-slave-drl0"
server_key: "89860330234c288ddeb4b1ce17d28dc5"
server_key_iv: "78f942581590fbbee95cd146859a1393"
server_key_version: "encryptionkey@0"
I20260808 04:34:49.229928 20172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:49.246336 20172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:49.250353 20172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:49.250631 20172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:49.251535 20172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:34:49.252686 20172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:49.252738 20172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:49.252789 20172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:49.252804 20172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:49.264819 20172 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:43787
I20260808 04:34:49.264850 20359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:43787 every 8 connection(s)
I20260808 04:34:49.265815 20172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:34:49.267143 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 20172
I20260808 04:34:49.267292 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:34:49.267618 16273 external_mini_cluster.cc:1428] Setting key a3ac291a096602a7f49e9be43df8a7ef
Aug 08 04:34:49 dist-test-slave-drl0 krb5kdc[19657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163689, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:49.281015 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:49.268094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:53285 (local address 127.15.228.126:33249)
0808 04:34:49.268358 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:49.268363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:49.269479 (+ 1116us) server_negotiation.cc:407] Connection header received
0808 04:34:49.270565 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:49.270568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:49.270624 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:49.270712 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:49.272018 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:49.272582 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:49.273737 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:49.274125 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:49.274244 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:49.276933 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:49.276937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:49.276940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:49.276966 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:49.278821 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:49.279391 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:49.279395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:49.279397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:49.279488 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:49.279852 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:49.279856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:49.279857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:49.280055 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:49.280167 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:49.280686 (+ 519us) server_negotiation.cc:299] Negotiation successful
0808 04:34:49.280815 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260808 04:34:49.281971 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:49.268512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:33249 (local address 127.15.228.67:53285)
0808 04:34:49.269323 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:49.269377 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:49.270356 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:49.270881 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:49.270887 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:49.271365 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:49.271868 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:49.271881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:49.272716 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:49.272719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:49.273611 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:49.273620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:49.274286 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:49.274290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:49.274469 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:49.275110 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:49.275127 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:49.276749 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:49.278990 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:49.278996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:49.279018 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:49.279274 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:49.279588 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:49.279591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:49.279593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:49.279737 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:49.280166 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:49.280173 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:49.280434 (+ 261us) client_negotiation.cc:769] Sending connection context
0808 04:34:49.280628 (+ 194us) client_negotiation.cc:240] Negotiation successful
0808 04:34:49.280993 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":144,"threads_started":1}
I20260808 04:34:49.283316 20360 heartbeater.cc:347] Connected to a master server at 127.15.228.126:33249
I20260808 04:34:49.283545 20360 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:49.284037 20360 heartbeater.cc:511] Master 127.15.228.126:33249 requested a full tablet report, sending...
I20260808 04:34:49.285174 19719 ts_manager.cc:194] Registered new tserver with Master: 4866071b23cf4b3eb5b975490c404f5c (127.15.228.67:43787)
I20260808 04:34:49.285851 19719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:53285
I20260808 04:34:49.296160 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 04:34:49.315673 19774 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:34:49.313699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59790 (local address 127.15.228.126:33249)
0808 04:34:49.313909 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:49.313913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:49.314105 (+ 192us) server_negotiation.cc:407] Connection header received
0808 04:34:49.314229 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:49.314233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:49.314299 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:49.314399 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:49.315447 (+ 1048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59790: BlockingRecv error: recv got EOF from 127.0.0.1:59790 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 04:34:49.316802 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 19810
I20260808 04:34:49.332302 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 19994
I20260808 04:34:49.344658 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 20172
I20260808 04:34:49.356096 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 19674
2026-08-08T04:34:49Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3759 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:34:49 dist-test-slave-drl0 krb5kdc[20379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:34:49 dist-test-slave-drl0 krb5kdc[20379](info): set up 2 sockets
Aug 08 04:34:49 dist-test-slave-drl0 krb5kdc[20379](info): commencing operation
krb5kdc: starting...
W20260808 04:34:50.653795 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.265s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:34:50 dist-test-slave-drl0 krb5kdc[20379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:34:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:34:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:34:50.799577 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:39787
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:38907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:39787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:50.933296 20397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:50.933633 20397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:50.933717 20397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:50.940614 20397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:34:50.940701 20397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:50.940724 20397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:50.940744 20397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:34:50.940768 20397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:34:50.948145 20397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:38907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:39787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:39787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:50.949435 20397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:50.950536 20397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:50.961272 20403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:50.961256 20410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:50.961261 20402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:50.961310 20397 server_base.cc:1034] running on GCE node
I20260808 04:34:50.962482 20397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:50.963825 20397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:50.965022 20397 hybrid_clock.cc:648] HybridClock initialized: now 1786163690964967 us; error 71 us; skew 500 ppm
Aug 08 04:34:50 dist-test-slave-drl0 krb5kdc[20379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163690, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:50.971352 20397 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:34:50.972345 20397 webserver.cc:460] Webserver started at http://127.15.228.126:46407/ using document root <none> and password file <none>
I20260808 04:34:50.972957 20397 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:50.973055 20397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:50.973301 20397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:50.975525 20397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "30ec5380f4834f48988b29e2e89606b5"
format_stamp: "Formatted at 2026-08-08 04:34:50 on dist-test-slave-drl0"
server_key: "2cd813646c94b4e08ec53fe77f6f4a09"
server_key_iv: "c4adb9e3a4ac39abe297607b748d1ec9"
server_key_version: "encryptionkey@0"
I20260808 04:34:50.976125 20397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "30ec5380f4834f48988b29e2e89606b5"
format_stamp: "Formatted at 2026-08-08 04:34:50 on dist-test-slave-drl0"
server_key: "2cd813646c94b4e08ec53fe77f6f4a09"
server_key_iv: "c4adb9e3a4ac39abe297607b748d1ec9"
server_key_version: "encryptionkey@0"
I20260808 04:34:50.980662 20397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:34:50.983884 20421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:50.985409 20397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:34:50.985607 20397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "30ec5380f4834f48988b29e2e89606b5"
format_stamp: "Formatted at 2026-08-08 04:34:50 on dist-test-slave-drl0"
server_key: "2cd813646c94b4e08ec53fe77f6f4a09"
server_key_iv: "c4adb9e3a4ac39abe297607b748d1ec9"
server_key_version: "encryptionkey@0"
I20260808 04:34:50.985735 20397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:51.004875 20397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:51.008641 20397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:51.008868 20397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:51.017274 20397 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:39787
I20260808 04:34:51.017629 20502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:39787 every 8 connection(s)
I20260808 04:34:51.018330 20397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:34:51.019932 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 20397
I20260808 04:34:51.020159 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:34:51.020812 16273 external_mini_cluster.cc:1428] Setting key 06f2394e46be9ecaa4ef15cd55456023
I20260808 04:34:51.022241 20503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:51.029255 20503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5: Bootstrap starting.
Aug 08 04:34:51 dist-test-slave-drl0 krb5kdc[20379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:51.031967 20503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:51.032972 20503 log.cc:824] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:51.035243 20503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5: No bootstrap required, opened a new log
I20260808 04:34:51.038542 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:51.022270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50176 (local address 127.15.228.126:39787)
0808 04:34:51.022991 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:51.023018 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:51.023098 (+ 80us) server_negotiation.cc:407] Connection header received
0808 04:34:51.024045 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:51.024067 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:51.024483 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:51.024844 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:51.025584 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:51.027101 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:51.028031 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:51.028713 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:51.028875 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:51.031689 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:51.031695 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:51.031711 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:51.031743 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:51.033894 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:51.034671 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:51.034677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:51.034682 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:51.034824 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:51.035457 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:51.035461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:51.035463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:51.035980 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:51.036180 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:51.036611 (+ 431us) server_negotiation.cc:299] Negotiation successful
0808 04:34:51.036916 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":202,"threads_started":1}
I20260808 04:34:51.040452 20503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ec5380f4834f48988b29e2e89606b5" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39787 } }
I20260808 04:34:51.040781 20503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:51.040841 20503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ec5380f4834f48988b29e2e89606b5, State: Initialized, Role: FOLLOWER
I20260808 04:34:51.041385 20503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ec5380f4834f48988b29e2e89606b5" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39787 } }
I20260808 04:34:51.041553 20503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:34:51.041610 20503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:34:51.041711 20503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:51.041909 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:34:51.042958 20503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ec5380f4834f48988b29e2e89606b5" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39787 } }
I20260808 04:34:51.043344 20503 leader_election.cc:302] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30ec5380f4834f48988b29e2e89606b5; no voters:
I20260808 04:34:51.043726 20503 leader_election.cc:288] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:34:51.043956 20511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:34:51.044232 20511 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5 [term 1 LEADER]: Becoming Leader. State: Replica: 30ec5380f4834f48988b29e2e89606b5, State: Running, Role: LEADER
I20260808 04:34:51.044550 20511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ec5380f4834f48988b29e2e89606b5" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39787 } }
I20260808 04:34:51.045064 20503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:34:51.046450 20512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30ec5380f4834f48988b29e2e89606b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ec5380f4834f48988b29e2e89606b5" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39787 } } }
I20260808 04:34:51.046593 20512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:34:51.046916 20514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30ec5380f4834f48988b29e2e89606b5. Latest consensus state: current_term: 1 leader_uuid: "30ec5380f4834f48988b29e2e89606b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ec5380f4834f48988b29e2e89606b5" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39787 } } }
I20260808 04:34:51.047048 20514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:34:51.047327 20528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:34:51.050688 20528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:34:51.057487 20528 catalog_manager.cc:1347] Generated new cluster ID: fabfbe52415b4c299ac5c3f990b1ad47
I20260808 04:34:51.057559 20528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:34:51.064057 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:51.059129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50190 (local address 127.15.228.126:39787)
0808 04:34:51.059268 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:51.059274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:51.059528 (+ 254us) server_negotiation.cc:407] Connection header received
0808 04:34:51.059694 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:51.059697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:51.059751 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:51.059839 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:51.060372 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:51.060922 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:51.061806 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:51.062051 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:51.062167 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:51.063101 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:51.063105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:51.063108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:34:51.063563 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:51.063595 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:34:51.063612 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:51.063721 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:51.063815 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 04:34:51.063878 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:34:51.082798 20528 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:34:51.083855 20528 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:34:51.089398 20528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30ec5380f4834f48988b29e2e89606b5: Generated new TSK 0
I20260808 04:34:51.090158 20528 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:34:51.098546 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 20397
2026-08-08T04:34:51Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1746 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:34:51 dist-test-slave-drl0 krb5kdc[20552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:34:51 dist-test-slave-drl0 krb5kdc[20552](info): set up 2 sockets
Aug 08 04:34:51 dist-test-slave-drl0 krb5kdc[20552](info): commencing operation
krb5kdc: starting...
W20260808 04:34:52.361452 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:34:52 dist-test-slave-drl0 krb5kdc[20552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:34:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:34:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:34:52.504173 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:42959
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:39211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:42959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:52.634087 20575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:52.634382 20575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:52.634431 20575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:52.638996 20575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:34:52.639065 20575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:52.639086 20575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:52.639103 20575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:34:52.639118 20575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:34:52.644665 20575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:39211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:42959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:42959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:52.645749 20575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:52.646780 20575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:52.656817 20585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:52.656814 20588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:52.656873 20591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:52.657037 20575 server_base.cc:1034] running on GCE node
I20260808 04:34:52.658003 20575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:52.659189 20575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:52.660499 20575 hybrid_clock.cc:648] HybridClock initialized: now 1786163692660494 us; error 50 us; skew 500 ppm
Aug 08 04:34:52 dist-test-slave-drl0 krb5kdc[20552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163692, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:52.663934 20575 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:34:52.665297 20575 webserver.cc:460] Webserver started at http://127.15.228.126:46671/ using document root <none> and password file <none>
I20260808 04:34:52.666092 20575 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:52.666174 20575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:52.666368 20575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:52.669404 20575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "3e87fca591a947efb8cdd03325265a62"
format_stamp: "Formatted at 2026-08-08 04:34:52 on dist-test-slave-drl0"
server_key: "c053d876df967f25f7c9f9ff5f4e4d8f"
server_key_iv: "955b264171331078f5c4249fbfc63066"
server_key_version: "encryptionkey@0"
I20260808 04:34:52.670176 20575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "3e87fca591a947efb8cdd03325265a62"
format_stamp: "Formatted at 2026-08-08 04:34:52 on dist-test-slave-drl0"
server_key: "c053d876df967f25f7c9f9ff5f4e4d8f"
server_key_iv: "955b264171331078f5c4249fbfc63066"
server_key_version: "encryptionkey@0"
I20260808 04:34:52.675313 20575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 04:34:52.678414 20602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:52.679883 20575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:34:52.680009 20575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "3e87fca591a947efb8cdd03325265a62"
format_stamp: "Formatted at 2026-08-08 04:34:52 on dist-test-slave-drl0"
server_key: "c053d876df967f25f7c9f9ff5f4e4d8f"
server_key_iv: "955b264171331078f5c4249fbfc63066"
server_key_version: "encryptionkey@0"
I20260808 04:34:52.680116 20575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:52.706861 20575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:52.711185 20575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:52.711364 20575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:52.720399 20575 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:42959
I20260808 04:34:52.720700 20673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:42959 every 8 connection(s)
I20260808 04:34:52.721470 20575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:34:52.722606 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 20575
I20260808 04:34:52.722757 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:34:52.723148 16273 external_mini_cluster.cc:1428] Setting key ea79f25cf5bc550fdde3d3d5756467a5
I20260808 04:34:52.724988 20675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:52.730680 20675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62: Bootstrap starting.
Aug 08 04:34:52 dist-test-slave-drl0 krb5kdc[20552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:52.733325 20675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:52.734215 20675 log.cc:824] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:52.736388 20675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62: No bootstrap required, opened a new log
I20260808 04:34:52.738570 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:52.724641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39718 (local address 127.15.228.126:42959)
0808 04:34:52.725278 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:52.725294 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:52.725331 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:34:52.726012 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:52.726028 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:52.726343 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:52.726676 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:52.727531 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:52.728531 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:52.729403 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:52.729873 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:52.730074 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:52.733352 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:52.733361 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:52.733411 (+ 50us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:52.733479 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:52.735653 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:52.736164 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:52.736177 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:52.736182 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:52.736316 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:52.736665 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:52.736669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:52.736671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:52.737072 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:52.737285 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:52.737570 (+ 285us) server_negotiation.cc:299] Negotiation successful
0808 04:34:52.737854 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":162,"threads_started":1}
I20260808 04:34:52.740396 20675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e87fca591a947efb8cdd03325265a62" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42959 } }
I20260808 04:34:52.740756 20675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:52.740918 20675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e87fca591a947efb8cdd03325265a62, State: Initialized, Role: FOLLOWER
I20260808 04:34:52.741429 20675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e87fca591a947efb8cdd03325265a62" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42959 } }
I20260808 04:34:52.741573 20675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:34:52.741617 20675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:34:52.741691 20675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:52.742883 20675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e87fca591a947efb8cdd03325265a62" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42959 } }
I20260808 04:34:52.743268 20675 leader_election.cc:302] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e87fca591a947efb8cdd03325265a62; no voters:
I20260808 04:34:52.743687 20675 leader_election.cc:288] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:34:52.743935 20683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:34:52.744207 20683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62 [term 1 LEADER]: Becoming Leader. State: Replica: 3e87fca591a947efb8cdd03325265a62, State: Running, Role: LEADER
I20260808 04:34:52.744516 20683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e87fca591a947efb8cdd03325265a62" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42959 } }
I20260808 04:34:52.745541 20675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:34:52.747015 20684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e87fca591a947efb8cdd03325265a62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e87fca591a947efb8cdd03325265a62" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42959 } } }
I20260808 04:34:52.747212 20684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62 [sys.catalog]: This master's current role is: LEADER
I20260808 04:34:52.747539 20685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e87fca591a947efb8cdd03325265a62. Latest consensus state: current_term: 1 leader_uuid: "3e87fca591a947efb8cdd03325265a62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e87fca591a947efb8cdd03325265a62" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42959 } } }
I20260808 04:34:52.747743 20685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62 [sys.catalog]: This master's current role is: LEADER
I20260808 04:34:52.749100 20694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:34:52.752221 20694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:34:52.758540 20694 catalog_manager.cc:1347] Generated new cluster ID: f5123adf58814fc7968ca47c47c30ebc
I20260808 04:34:52.758617 20694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:34:52.767213 20694 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:34:52.768144 20694 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:34:52.780582 20694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e87fca591a947efb8cdd03325265a62: Generated new TSK 0
I20260808 04:34:52.781348 20694 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
I20260808 04:34:52.816545 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:42959
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:39211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:52.943149 20709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:52.943459 20709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:52.943537 20709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:52.948455 20709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:52.948557 20709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:52.948649 20709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:34:52.954244 20709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:39211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:42959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:52.955474 20709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:52.956565 20709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:52.967095 20717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:52.967128 20720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:52.967092 20716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:52.967561 20709 server_base.cc:1034] running on GCE node
I20260808 04:34:52.968139 20709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:52.968804 20709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:52.970028 20709 hybrid_clock.cc:648] HybridClock initialized: now 1786163692969993 us; error 49 us; skew 500 ppm
Aug 08 04:34:52 dist-test-slave-drl0 krb5kdc[20552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163692, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:52.973217 20709 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:34:52.974184 20709 webserver.cc:460] Webserver started at http://127.15.228.65:35461/ using document root <none> and password file <none>
I20260808 04:34:52.974768 20709 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:52.974870 20709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:52.975131 20709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:52.977195 20709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "3eb661e813a543e6b23e80435e5ca911"
format_stamp: "Formatted at 2026-08-08 04:34:52 on dist-test-slave-drl0"
server_key: "88f146655cb10dbe4bac502f0c70767f"
server_key_iv: "ffb09806389986a5d2685cfa7be57e3c"
server_key_version: "encryptionkey@0"
I20260808 04:34:52.977777 20709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "3eb661e813a543e6b23e80435e5ca911"
format_stamp: "Formatted at 2026-08-08 04:34:52 on dist-test-slave-drl0"
server_key: "88f146655cb10dbe4bac502f0c70767f"
server_key_iv: "ffb09806389986a5d2685cfa7be57e3c"
server_key_version: "encryptionkey@0"
I20260808 04:34:52.982566 20709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 04:34:52.986541 20733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:52.988184 20709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:34:52.988361 20709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "3eb661e813a543e6b23e80435e5ca911"
format_stamp: "Formatted at 2026-08-08 04:34:52 on dist-test-slave-drl0"
server_key: "88f146655cb10dbe4bac502f0c70767f"
server_key_iv: "ffb09806389986a5d2685cfa7be57e3c"
server_key_version: "encryptionkey@0"
I20260808 04:34:52.988497 20709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:53.014652 20709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:53.018862 20709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:53.019244 20709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:53.020200 20709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:34:53.021405 20709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:53.021466 20709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:53.021519 20709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:53.021535 20709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:53.032482 20709 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:41989
I20260808 04:34:53.032773 20902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:41989 every 8 connection(s)
I20260808 04:34:53.033494 20709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:34:53.037528 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 20709
I20260808 04:34:53.037703 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:34:53.038092 16273 external_mini_cluster.cc:1428] Setting key a2db6c4f769b279461867a05265a5c55
Aug 08 04:34:53 dist-test-slave-drl0 krb5kdc[20552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163692, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.228.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.66@KRBTEST.COM" created.
I20260808 04:34:53.052583 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:53.035962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:40379 (local address 127.15.228.126:42959)
0808 04:34:53.036167 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:53.036173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:53.037029 (+ 856us) server_negotiation.cc:407] Connection header received
0808 04:34:53.039114 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:53.039119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:53.039321 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:53.039483 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:53.041410 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:53.042141 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:53.043830 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:53.044129 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:53.044270 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:53.047585 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:53.047589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:53.047591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:53.047617 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:53.049387 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:53.050053 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:53.050056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:53.050057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:53.050137 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:53.050711 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:53.050714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:53.050715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:53.050990 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:53.051087 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:53.051939 (+ 852us) server_negotiation.cc:299] Negotiation successful
0808 04:34:53.052242 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 04:34:53.053210 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:53.036338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:42959 (local address 127.15.228.65:40379)
0808 04:34:53.036847 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:53.036905 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:53.038705 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:53.039876 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:53.039887 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:53.040456 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:53.041189 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:53.041210 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:53.042351 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:53.042354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:53.043629 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:53.043648 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:53.044321 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:53.044326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:53.044524 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:53.045188 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:53.045213 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:53.047314 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:53.049630 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:53.049635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:53.049648 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:53.049923 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:53.050382 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:53.050388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:53.050393 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:53.050576 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:53.051122 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:53.051128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:53.051465 (+ 337us) client_negotiation.cc:769] Sending connection context
0808 04:34:53.051709 (+ 244us) client_negotiation.cc:240] Negotiation successful
0808 04:34:53.052005 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":121,"threads_started":1}
I20260808 04:34:53.054867 20903 heartbeater.cc:347] Connected to a master server at 127.15.228.126:42959
I20260808 04:34:53.055295 20903 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:53.055889 20903 heartbeater.cc:511] Master 127.15.228.126:42959 requested a full tablet report, sending...
I20260808 04:34:53.057700 20618 ts_manager.cc:194] Registered new tserver with Master: 3eb661e813a543e6b23e80435e5ca911 (127.15.228.65:41989)
I20260808 04:34:53.059443 20618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:40379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
I20260808 04:34:53.074510 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:42959
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:39211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:53.210480 20909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:53.210968 20909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:53.211032 20909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:53.217497 20909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:53.217631 20909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:53.217777 20909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:34:53.224891 20909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:39211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:42959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:53.226014 20909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:53.227079 20909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:53.239733 20919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:53.239733 20917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:53.239722 20921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:53.239835 20909 server_base.cc:1034] running on GCE node
I20260808 04:34:53.240872 20909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:53.241770 20909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:53.243126 20909 hybrid_clock.cc:648] HybridClock initialized: now 1786163693243082 us; error 57 us; skew 500 ppm
Aug 08 04:34:53 dist-test-slave-drl0 krb5kdc[20552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163693, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:53.246284 20909 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:34:53.247464 20909 webserver.cc:460] Webserver started at http://127.15.228.66:41163/ using document root <none> and password file <none>
I20260808 04:34:53.248114 20909 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:53.248211 20909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:53.248493 20909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:53.250806 20909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "883a130c49304da2837b7525178c7b18"
format_stamp: "Formatted at 2026-08-08 04:34:53 on dist-test-slave-drl0"
server_key: "43f19f08a71d69ebb47e219ab988e5c5"
server_key_iv: "4e6a27b0349fc8977771ee1a2ca840cc"
server_key_version: "encryptionkey@0"
I20260808 04:34:53.251608 20909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "883a130c49304da2837b7525178c7b18"
format_stamp: "Formatted at 2026-08-08 04:34:53 on dist-test-slave-drl0"
server_key: "43f19f08a71d69ebb47e219ab988e5c5"
server_key_iv: "4e6a27b0349fc8977771ee1a2ca840cc"
server_key_version: "encryptionkey@0"
I20260808 04:34:53.256316 20909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:34:53.259192 20931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:53.260792 20909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:34:53.260979 20909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "883a130c49304da2837b7525178c7b18"
format_stamp: "Formatted at 2026-08-08 04:34:53 on dist-test-slave-drl0"
server_key: "43f19f08a71d69ebb47e219ab988e5c5"
server_key_iv: "4e6a27b0349fc8977771ee1a2ca840cc"
server_key_version: "encryptionkey@0"
I20260808 04:34:53.261126 20909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:53.283140 20909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:53.288789 20909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:53.289393 20909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:53.290534 20909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:34:53.292033 20909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:53.292109 20909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:53.292162 20909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:53.292181 20909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:53.303854 20909 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:34761
I20260808 04:34:53.303951 21096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:34761 every 8 connection(s)
I20260808 04:34:53.304879 20909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:34:53.314312 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 20909
I20260808 04:34:53.314467 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:34:53.314849 16273 external_mini_cluster.cc:1428] Setting key 69dbb5228d3743c19e540bb093a2cfef
Aug 08 04:34:53 dist-test-slave-drl0 krb5kdc[20552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163693, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:53.322722 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:53.307534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:43931 (local address 127.15.228.126:42959)
0808 04:34:53.307743 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:53.307747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:53.308505 (+ 758us) server_negotiation.cc:407] Connection header received
0808 04:34:53.309965 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:53.309968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:53.310018 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:53.310097 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:53.311372 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:53.311928 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:53.313143 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:53.313486 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:53.313608 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:53.317430 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:53.317433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:53.317435 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:53.317460 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:53.319597 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:53.321026 (+ 1429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:53.321029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:53.321030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:53.321114 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:53.321528 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:53.321535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:53.321538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:53.321740 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:53.321860 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:53.322399 (+ 539us) server_negotiation.cc:299] Negotiation successful
0808 04:34:53.322543 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260808 04:34:53.323654 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:53.307846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:42959 (local address 127.15.228.66:43931)
0808 04:34:53.308343 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:53.308387 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:53.309726 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:53.310316 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:53.310322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:53.310682 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:53.311216 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:53.311231 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:53.312073 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:53.312076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:53.312994 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:53.313005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:53.313744 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:53.313754 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:53.313923 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:53.314870 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:53.314894 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:53.317210 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:53.320004 (+ 2794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:53.320012 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:53.320140 (+ 128us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:53.320856 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:53.321243 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:53.321247 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:53.321249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:53.321412 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:53.321860 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:53.321868 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:53.322149 (+ 281us) client_negotiation.cc:769] Sending connection context
0808 04:34:53.322356 (+ 207us) client_negotiation.cc:240] Negotiation successful
0808 04:34:53.322719 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":111,"threads_started":1}
I20260808 04:34:53.325316 21097 heartbeater.cc:347] Connected to a master server at 127.15.228.126:42959
I20260808 04:34:53.325621 21097 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:53.326339 21097 heartbeater.cc:511] Master 127.15.228.126:42959 requested a full tablet report, sending...
I20260808 04:34:53.327960 20618 ts_manager.cc:194] Registered new tserver with Master: 883a130c49304da2837b7525178c7b18 (127.15.228.66:34761)
WARNING: no policy specified for kudu/127.15.228.67@KRBTEST.COM; defaulting to no policy
I20260808 04:34:53.328856 20618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:43931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
I20260808 04:34:53.352728 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:42959
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:39211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:53.484376 21111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:53.484948 21111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:53.485077 21111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:53.491914 21111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:53.492175 21111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:53.492374 21111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:34:53.500332 21111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:39211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:42959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:53.501883 21111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:53.503494 21111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:53.515277 21127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:53.515358 21124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:53.515277 21121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:53.515504 21111 server_base.cc:1034] running on GCE node
I20260808 04:34:53.516017 21111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:53.516698 21111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:53.517936 21111 hybrid_clock.cc:648] HybridClock initialized: now 1786163693517919 us; error 51 us; skew 500 ppm
Aug 08 04:34:53 dist-test-slave-drl0 krb5kdc[20552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163693, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:53.520968 21111 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:34:53.521973 21111 webserver.cc:460] Webserver started at http://127.15.228.67:34903/ using document root <none> and password file <none>
I20260808 04:34:53.522521 21111 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:53.522583 21111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:53.522754 21111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:53.524976 21111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "47c042f03de64b3094227cf3eb4960ae"
format_stamp: "Formatted at 2026-08-08 04:34:53 on dist-test-slave-drl0"
server_key: "4855c1efb545d563a7a8476042d1bc6f"
server_key_iv: "ac438477c3cf7bb6861ead7013c06511"
server_key_version: "encryptionkey@0"
I20260808 04:34:53.525513 21111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "47c042f03de64b3094227cf3eb4960ae"
format_stamp: "Formatted at 2026-08-08 04:34:53 on dist-test-slave-drl0"
server_key: "4855c1efb545d563a7a8476042d1bc6f"
server_key_iv: "ac438477c3cf7bb6861ead7013c06511"
server_key_version: "encryptionkey@0"
I20260808 04:34:53.530827 21111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 04:34:53.533968 21137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:53.535563 21111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:34:53.535681 21111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "47c042f03de64b3094227cf3eb4960ae"
format_stamp: "Formatted at 2026-08-08 04:34:53 on dist-test-slave-drl0"
server_key: "4855c1efb545d563a7a8476042d1bc6f"
server_key_iv: "ac438477c3cf7bb6861ead7013c06511"
server_key_version: "encryptionkey@0"
I20260808 04:34:53.535787 21111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:53.573941 21111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:53.577457 21111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:53.577630 21111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:53.578197 21111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:34:53.579051 21111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:53.579110 21111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:53.579152 21111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:53.579169 21111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:53.590809 21111 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:36255
I20260808 04:34:53.590831 21295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:36255 every 8 connection(s)
I20260808 04:34:53.591775 21111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:34:53.593521 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 21111
I20260808 04:34:53.593815 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:34:53.594247 16273 external_mini_cluster.cc:1428] Setting key 627febc59f6fff498d826d4a68fb9645
Aug 08 04:34:53 dist-test-slave-drl0 krb5kdc[20552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163693, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:53.609616 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:53.594717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:38845 (local address 127.15.228.126:42959)
0808 04:34:53.595160 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:53.595167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:53.595737 (+ 570us) server_negotiation.cc:407] Connection header received
0808 04:34:53.597312 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:53.597314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:53.597362 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:53.597440 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:53.598578 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:53.599148 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:53.600441 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:53.600682 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:53.600784 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:53.604148 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:53.604154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:53.604156 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:53.604196 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:53.605968 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:53.606726 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:53.606729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:53.606731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:53.606855 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:53.607448 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:53.607454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:53.607456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:53.607797 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:53.607949 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:53.609274 (+ 1325us) server_negotiation.cc:299] Negotiation successful
0808 04:34:53.609435 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 04:34:53.610371 21298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:53.594979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:42959 (local address 127.15.228.67:38845)
0808 04:34:53.595569 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:53.595612 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:53.597104 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:53.597565 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:53.597571 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:53.597933 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:53.598409 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:53.598424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:53.599303 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:53.599306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:53.600213 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:53.600224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:53.600877 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:53.600880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:53.601044 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:53.601624 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:53.601641 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:53.603750 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:53.606259 (+ 2509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:53.606265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:53.606279 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:53.606587 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:53.607001 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:53.607005 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:53.607007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:53.607236 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:53.608000 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:53.608009 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:53.608705 (+ 696us) client_negotiation.cc:769] Sending connection context
0808 04:34:53.609097 (+ 392us) client_negotiation.cc:240] Negotiation successful
0808 04:34:53.609447 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":162,"threads_started":1}
I20260808 04:34:53.611867 21296 heartbeater.cc:347] Connected to a master server at 127.15.228.126:42959
I20260808 04:34:53.612092 21296 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:53.612588 21296 heartbeater.cc:511] Master 127.15.228.126:42959 requested a full tablet report, sending...
I20260808 04:34:53.613870 20618 ts_manager.cc:194] Registered new tserver with Master: 47c042f03de64b3094227cf3eb4960ae (127.15.228.67:36255)
I20260808 04:34:53.614655 20618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:38845
I20260808 04:34:53.622292 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 04:34:53 dist-test-slave-drl0 krb5kdc[20552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163693, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 04:34:53 dist-test-slave-drl0 krb5kdc[20552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163693, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:53.657716 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:53.646045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60382 (local address 127.15.228.126:42959)
0808 04:34:53.646190 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:53.646194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:53.646383 (+ 189us) server_negotiation.cc:407] Connection header received
0808 04:34:53.646484 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:53.646486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:53.646536 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:53.646608 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:53.648254 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:53.648925 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:53.649903 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:53.650126 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:53.650219 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:53.653611 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:53.653615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:53.653617 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:53.653643 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:53.655241 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:53.656030 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:53.656044 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:53.656045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:53.656177 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:53.656664 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:53.656671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:53.656674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:53.656947 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:53.657114 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:53.657384 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 04:34:53.657545 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 04:34:53.658203 20618 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60382
I20260808 04:34:53.659644 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 20709
I20260808 04:34:53.682771 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 20909
I20260808 04:34:53.697646 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 21111
I20260808 04:34:53.717976 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 20575
2026-08-08T04:34:53Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2612 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:34:53 dist-test-slave-drl0 krb5kdc[21313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:34:53 dist-test-slave-drl0 krb5kdc[21313](info): set up 2 sockets
Aug 08 04:34:53 dist-test-slave-drl0 krb5kdc[21313](info): commencing operation
krb5kdc: starting...
W20260808 04:34:54.978547 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.232s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:34:55 dist-test-slave-drl0 krb5kdc[21313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:34:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:34:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:34:55.126550 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:45977
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:45087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:45977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:55.259756 21330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:55.260448 21330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:55.260722 21330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:55.265985 21330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:34:55.266052 21330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:55.266072 21330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:55.266090 21330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:34:55.266106 21330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:34:55.272123 21330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:45087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:45977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:45977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:55.273252 21330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:55.274317 21330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:34:55.283308 21330 server_base.cc:1034] running on GCE node
W20260808 04:34:55.283273 21339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:55.283289 21341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:55.283473 21336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:55.284076 21330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:55.285197 21330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:55.286407 21330 hybrid_clock.cc:648] HybridClock initialized: now 1786163695286376 us; error 45 us; skew 500 ppm
Aug 08 04:34:55 dist-test-slave-drl0 krb5kdc[21313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163695, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:55.289610 21330 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:34:55.291152 21330 webserver.cc:460] Webserver started at http://127.15.228.126:45067/ using document root <none> and password file <none>
I20260808 04:34:55.291863 21330 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:55.291972 21330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:55.292423 21330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:55.294621 21330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "1e1d0e7e225f457883befb6960d81c58"
format_stamp: "Formatted at 2026-08-08 04:34:55 on dist-test-slave-drl0"
server_key: "8d58804de5acc33b84e375b58f4c5d2b"
server_key_iv: "cd051d4548bc45788b1f3ba754a0b685"
server_key_version: "encryptionkey@0"
I20260808 04:34:55.295276 21330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "1e1d0e7e225f457883befb6960d81c58"
format_stamp: "Formatted at 2026-08-08 04:34:55 on dist-test-slave-drl0"
server_key: "8d58804de5acc33b84e375b58f4c5d2b"
server_key_iv: "cd051d4548bc45788b1f3ba754a0b685"
server_key_version: "encryptionkey@0"
I20260808 04:34:55.300542 21330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260808 04:34:55.303516 21350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:55.305150 21330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:34:55.305260 21330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "1e1d0e7e225f457883befb6960d81c58"
format_stamp: "Formatted at 2026-08-08 04:34:55 on dist-test-slave-drl0"
server_key: "8d58804de5acc33b84e375b58f4c5d2b"
server_key_iv: "cd051d4548bc45788b1f3ba754a0b685"
server_key_version: "encryptionkey@0"
I20260808 04:34:55.305356 21330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:55.318789 21330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:55.322326 21330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:55.322639 21330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:55.331717 21330 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:45977
I20260808 04:34:55.331748 21428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:45977 every 8 connection(s)
I20260808 04:34:55.332886 21330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:34:55.335847 21429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:55.336319 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 21330
I20260808 04:34:55.336470 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:34:55.336922 16273 external_mini_cluster.cc:1428] Setting key a772aa67cf86e911aec95f9fa5667701
I20260808 04:34:55.342764 21429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58: Bootstrap starting.
I20260808 04:34:55.345572 21429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:55.346560 21429 log.cc:824] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:34:55 dist-test-slave-drl0 krb5kdc[21313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:55.348867 21429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58: No bootstrap required, opened a new log
I20260808 04:34:55.352566 21429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1d0e7e225f457883befb6960d81c58" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45977 } }
I20260808 04:34:55.352818 21429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:55.352900 21429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e1d0e7e225f457883befb6960d81c58, State: Initialized, Role: FOLLOWER
I20260808 04:34:55.353430 21429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1d0e7e225f457883befb6960d81c58" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45977 } }
I20260808 04:34:55.353612 21429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:34:55.353670 21429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:34:55.353757 21429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:55.354565 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:55.339176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53358 (local address 127.15.228.126:45977)
0808 04:34:55.339740 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:55.339762 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:55.339813 (+ 51us) server_negotiation.cc:407] Connection header received
0808 04:34:55.340649 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:55.340670 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:55.341107 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:55.341485 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:55.342551 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:55.343699 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:55.344926 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:55.345409 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:55.345548 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:55.348503 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:55.348508 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:55.348524 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:55.348552 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:55.351290 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:55.351827 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:55.351832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:55.351837 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:55.351956 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:55.352245 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:55.352249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:55.352251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:55.352669 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:55.352903 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:55.353222 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 04:34:55.353534 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":179,"threads_started":1}
I20260808 04:34:55.355048 21429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1d0e7e225f457883befb6960d81c58" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45977 } }
I20260808 04:34:55.355394 21429 leader_election.cc:302] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e1d0e7e225f457883befb6960d81c58; no voters:
I20260808 04:34:55.355736 21429 leader_election.cc:288] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:34:55.355871 21436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:34:55.356107 21436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58 [term 1 LEADER]: Becoming Leader. State: Replica: 1e1d0e7e225f457883befb6960d81c58, State: Running, Role: LEADER
I20260808 04:34:55.356519 21436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1d0e7e225f457883befb6960d81c58" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45977 } }
I20260808 04:34:55.357074 21429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:34:55.358362 21438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e1d0e7e225f457883befb6960d81c58. Latest consensus state: current_term: 1 leader_uuid: "1e1d0e7e225f457883befb6960d81c58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1d0e7e225f457883befb6960d81c58" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45977 } } }
I20260808 04:34:55.358366 21437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e1d0e7e225f457883befb6960d81c58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1d0e7e225f457883befb6960d81c58" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45977 } } }
I20260808 04:34:55.358572 21438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58 [sys.catalog]: This master's current role is: LEADER
I20260808 04:34:55.358621 21437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58 [sys.catalog]: This master's current role is: LEADER
I20260808 04:34:55.359639 21452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:34:55.363539 21452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:34:55.370219 21452 catalog_manager.cc:1347] Generated new cluster ID: cd150d238ccb4eb6b4f2f8da434020e2
I20260808 04:34:55.370357 21452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:34:55.391007 21452 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:34:55.393301 21452 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:34:55.407212 21452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e1d0e7e225f457883befb6960d81c58: Generated new TSK 0
I20260808 04:34:55.408025 21452 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
I20260808 04:34:55.446697 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:45977
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:45087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:55.575451 21469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:55.575748 21469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:55.575829 21469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:55.580201 21469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:55.580313 21469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:55.580410 21469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:34:55.586261 21469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:45087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:55.587467 21469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:55.588575 21469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:55.599331 21476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:55.599315 21482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:55.599308 21477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:55.599350 21469 server_base.cc:1034] running on GCE node
I20260808 04:34:55.599972 21469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:55.600648 21469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:55.601862 21469 hybrid_clock.cc:648] HybridClock initialized: now 1786163695601828 us; error 47 us; skew 500 ppm
Aug 08 04:34:55 dist-test-slave-drl0 krb5kdc[21313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163695, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:55.605434 21469 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:34:55.606739 21469 webserver.cc:460] Webserver started at http://127.15.228.65:41045/ using document root <none> and password file <none>
I20260808 04:34:55.607458 21469 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:55.607564 21469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:55.607930 21469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:55.610280 21469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "9d49261fceb64b72b50f04d08addf484"
format_stamp: "Formatted at 2026-08-08 04:34:55 on dist-test-slave-drl0"
server_key: "e07ad24fffd2027fde6b09d4fbd32842"
server_key_iv: "f7ccf9f4342b580483d8f0acddb92959"
server_key_version: "encryptionkey@0"
I20260808 04:34:55.610898 21469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "9d49261fceb64b72b50f04d08addf484"
format_stamp: "Formatted at 2026-08-08 04:34:55 on dist-test-slave-drl0"
server_key: "e07ad24fffd2027fde6b09d4fbd32842"
server_key_iv: "f7ccf9f4342b580483d8f0acddb92959"
server_key_version: "encryptionkey@0"
I20260808 04:34:55.615000 21469 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:34:55.617660 21492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:55.619714 21469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:34:55.619952 21469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "9d49261fceb64b72b50f04d08addf484"
format_stamp: "Formatted at 2026-08-08 04:34:55 on dist-test-slave-drl0"
server_key: "e07ad24fffd2027fde6b09d4fbd32842"
server_key_iv: "f7ccf9f4342b580483d8f0acddb92959"
server_key_version: "encryptionkey@0"
I20260808 04:34:55.620209 21469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:55.631067 21469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:55.634903 21469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:55.635142 21469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:55.635787 21469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:34:55.636777 21469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:55.636859 21469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:55.636931 21469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:55.636986 21469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:55.648290 21469 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:35913
I20260808 04:34:55.648325 21654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:35913 every 8 connection(s)
I20260808 04:34:55.649241 21469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:34:55.655150 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 21469
I20260808 04:34:55.655294 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:34:55.655617 16273 external_mini_cluster.cc:1428] Setting key ca50f865d5f82855f44123fed1f90268
Aug 08 04:34:55 dist-test-slave-drl0 krb5kdc[21313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163695, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.228.66@KRBTEST.COM; defaulting to no policy
I20260808 04:34:55.668218 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:55.651891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:52113 (local address 127.15.228.126:45977)
0808 04:34:55.652083 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:55.652087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:55.652951 (+ 864us) server_negotiation.cc:407] Connection header received
0808 04:34:55.654968 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:55.654973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:55.655069 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:55.655209 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:55.657265 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:55.658263 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:55.660246 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:55.660699 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:55.660848 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:55.663790 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:55.663797 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:55.663801 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:55.663838 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:55.665711 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:55.666256 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:55.666260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:55.666262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:55.666356 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:55.666766 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:55.666779 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:55.666782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:55.667130 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:55.667268 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:55.667898 (+ 630us) server_negotiation.cc:299] Negotiation successful
0808 04:34:55.668036 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260808 04:34:55.669423 21660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:55.652259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45977 (local address 127.15.228.65:52113)
0808 04:34:55.652775 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:55.652819 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:55.654488 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:55.655463 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:55.655473 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:55.655983 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:55.656980 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:55.657005 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:55.658444 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:55.658451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:55.660080 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:55.660092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:55.660902 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:55.660905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:55.661081 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:55.661733 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:55.661750 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:55.663593 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:55.665867 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:55.665872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:55.665885 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:55.666139 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:55.666463 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:55.666465 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:55.666466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:55.666601 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:55.667283 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:55.667289 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:55.667544 (+ 255us) client_negotiation.cc:769] Sending connection context
0808 04:34:55.667844 (+ 300us) client_negotiation.cc:240] Negotiation successful
0808 04:34:55.668192 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":124,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.66@KRBTEST.COM" created.
I20260808 04:34:55.671002 21656 heartbeater.cc:347] Connected to a master server at 127.15.228.126:45977
I20260808 04:34:55.671264 21656 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:55.672276 21656 heartbeater.cc:511] Master 127.15.228.126:45977 requested a full tablet report, sending...
I20260808 04:34:55.674043 21369 ts_manager.cc:194] Registered new tserver with Master: 9d49261fceb64b72b50f04d08addf484 (127.15.228.65:35913)
I20260808 04:34:55.675276 21369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:52113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
I20260808 04:34:55.691599 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:45977
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:45087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:55.822081 21664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:55.822377 21664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:55.822448 21664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:55.827612 21664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:55.827701 21664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:55.827788 21664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:34:55.833366 21664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:45087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:55.834661 21664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:55.835831 21664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:55.846874 21672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:55.847077 21664 server_base.cc:1034] running on GCE node
W20260808 04:34:55.846982 21669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:55.847260 21670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:55.847910 21664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:55.848958 21664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:55.850188 21664 hybrid_clock.cc:648] HybridClock initialized: now 1786163695850164 us; error 36 us; skew 500 ppm
Aug 08 04:34:55 dist-test-slave-drl0 krb5kdc[21313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163695, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:55.853806 21664 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:34:55.854911 21664 webserver.cc:460] Webserver started at http://127.15.228.66:34295/ using document root <none> and password file <none>
I20260808 04:34:55.855611 21664 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:55.855691 21664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:55.855918 21664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:55.858213 21664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "f1e5349f7c6c448997931854bd71f4ee"
format_stamp: "Formatted at 2026-08-08 04:34:55 on dist-test-slave-drl0"
server_key: "72f3be53a8a2d42d14160f31fee8ef48"
server_key_iv: "36ceca68191a080a88ed17285593ecd3"
server_key_version: "encryptionkey@0"
I20260808 04:34:55.859149 21664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "f1e5349f7c6c448997931854bd71f4ee"
format_stamp: "Formatted at 2026-08-08 04:34:55 on dist-test-slave-drl0"
server_key: "72f3be53a8a2d42d14160f31fee8ef48"
server_key_iv: "36ceca68191a080a88ed17285593ecd3"
server_key_version: "encryptionkey@0"
I20260808 04:34:55.863451 21664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:34:55.866349 21684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:55.867924 21664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:34:55.868077 21664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "f1e5349f7c6c448997931854bd71f4ee"
format_stamp: "Formatted at 2026-08-08 04:34:55 on dist-test-slave-drl0"
server_key: "72f3be53a8a2d42d14160f31fee8ef48"
server_key_iv: "36ceca68191a080a88ed17285593ecd3"
server_key_version: "encryptionkey@0"
I20260808 04:34:55.868197 21664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:55.903682 21664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:55.908258 21664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:55.908497 21664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:55.909309 21664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:34:55.910238 21664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:55.910288 21664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:55.910329 21664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:55.910346 21664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:55.922546 21664 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:38695
I20260808 04:34:55.922636 21852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:38695 every 8 connection(s)
I20260808 04:34:55.923966 21664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:34:55.932031 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 21664
I20260808 04:34:55.932253 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:34:55.932953 16273 external_mini_cluster.cc:1428] Setting key 58d994798288fe073e3c251bd4c2c562
Aug 08 04:34:55 dist-test-slave-drl0 krb5kdc[21313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163695, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:55.941097 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:55.926433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:54177 (local address 127.15.228.126:45977)
0808 04:34:55.926640 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:55.926644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:55.927564 (+ 920us) server_negotiation.cc:407] Connection header received
0808 04:34:55.929137 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:55.929140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:55.929202 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:55.929310 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:55.930551 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:55.931158 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:55.932704 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:55.933086 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:55.933262 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:55.936230 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:55.936237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:55.936243 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:55.936277 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:55.938074 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:55.939202 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:55.939207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:55.939208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:55.939346 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:55.939893 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:55.939896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:55.939899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:55.940104 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:55.940229 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:55.940776 (+ 547us) server_negotiation.cc:299] Negotiation successful
0808 04:34:55.940904 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260808 04:34:55.942749 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:55.926823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45977 (local address 127.15.228.66:54177)
0808 04:34:55.927410 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:55.927456 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:55.928922 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:55.929486 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:55.929493 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:55.929885 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:55.930406 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:55.930420 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:55.931290 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:55.931293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:55.932195 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:55.932205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:55.933352 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:55.933356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:55.933523 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:55.934278 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:55.934295 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:55.936054 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:55.938287 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:55.938302 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:55.938334 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:55.938999 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:55.939506 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:55.939509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:55.939511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:55.939738 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:55.940229 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:55.940237 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:55.940508 (+ 271us) client_negotiation.cc:769] Sending connection context
0808 04:34:55.940715 (+ 207us) client_negotiation.cc:240] Negotiation successful
0808 04:34:55.941104 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":144,"threads_started":1}
I20260808 04:34:55.944406 21854 heartbeater.cc:347] Connected to a master server at 127.15.228.126:45977
I20260808 04:34:55.944732 21854 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:55.945433 21854 heartbeater.cc:511] Master 127.15.228.126:45977 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.228.67@KRBTEST.COM; defaulting to no policy
I20260808 04:34:55.946738 21369 ts_manager.cc:194] Registered new tserver with Master: f1e5349f7c6c448997931854bd71f4ee (127.15.228.66:38695)
I20260808 04:34:55.947573 21369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:54177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
I20260808 04:34:55.972270 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:45977
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:45087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:56.106408 21865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:56.107260 21865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:56.107412 21865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:56.112310 21865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:56.112372 21865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:56.112437 21865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:34:56.118541 21865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:45087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:56.119761 21865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:56.120868 21865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:56.132652 21870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:56.132699 21871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:56.132682 21873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:56.132961 21865 server_base.cc:1034] running on GCE node
I20260808 04:34:56.133695 21865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:56.134430 21865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:56.135754 21865 hybrid_clock.cc:648] HybridClock initialized: now 1786163696135725 us; error 58 us; skew 500 ppm
Aug 08 04:34:56 dist-test-slave-drl0 krb5kdc[21313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163696, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:56.143203 21865 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:34:56.144423 21865 webserver.cc:460] Webserver started at http://127.15.228.67:34135/ using document root <none> and password file <none>
I20260808 04:34:56.145076 21865 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:56.145200 21865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:56.145463 21865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:56.147770 21865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "581c503917544a6b98d9e0f5af3d648d"
format_stamp: "Formatted at 2026-08-08 04:34:56 on dist-test-slave-drl0"
server_key: "9dda29e0db05aaca1ee6fd6b9f21df90"
server_key_iv: "9977975f57e9ecbbc473547b09fb7241"
server_key_version: "encryptionkey@0"
I20260808 04:34:56.148373 21865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "581c503917544a6b98d9e0f5af3d648d"
format_stamp: "Formatted at 2026-08-08 04:34:56 on dist-test-slave-drl0"
server_key: "9dda29e0db05aaca1ee6fd6b9f21df90"
server_key_iv: "9977975f57e9ecbbc473547b09fb7241"
server_key_version: "encryptionkey@0"
I20260808 04:34:56.152966 21865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:34:56.156059 21881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:56.157796 21865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:34:56.157953 21865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "581c503917544a6b98d9e0f5af3d648d"
format_stamp: "Formatted at 2026-08-08 04:34:56 on dist-test-slave-drl0"
server_key: "9dda29e0db05aaca1ee6fd6b9f21df90"
server_key_iv: "9977975f57e9ecbbc473547b09fb7241"
server_key_version: "encryptionkey@0"
I20260808 04:34:56.158077 21865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:56.175503 21865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:56.179335 21865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:56.179554 21865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:56.180208 21865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:34:56.181130 21865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:56.181203 21865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:56.181313 21865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:56.181356 21865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:56.193895 21865 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:42893
I20260808 04:34:56.194015 22038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:42893 every 8 connection(s)
I20260808 04:34:56.195263 21865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:34:56.203488 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 21865
I20260808 04:34:56.203621 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:34:56.204063 16273 external_mini_cluster.cc:1428] Setting key b7f003caf12f80e034ccd741b50bf5ba
Aug 08 04:34:56 dist-test-slave-drl0 krb5kdc[21313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163696, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:56.214376 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:56.197839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:36185 (local address 127.15.228.126:45977)
0808 04:34:56.198030 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:56.198035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:56.199156 (+ 1121us) server_negotiation.cc:407] Connection header received
0808 04:34:56.200664 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:56.200668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:56.200731 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:56.200845 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:56.202325 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:56.203027 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:56.204889 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:56.205405 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:56.205564 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:56.209621 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:56.209625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:56.209627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:56.209657 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:56.211734 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:56.212447 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:56.212451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:56.212452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:56.212556 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:56.212981 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:56.212984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:56.212985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:56.213216 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:56.213311 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:56.213982 (+ 671us) server_negotiation.cc:299] Negotiation successful
0808 04:34:56.214207 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 04:34:56.215207 22041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:56.198324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45977 (local address 127.15.228.67:36185)
0808 04:34:56.198966 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:56.199015 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:56.200422 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:56.201042 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:56.201049 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:56.201447 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:56.201974 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:56.201988 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:56.203248 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:56.203251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:56.204679 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:56.204699 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:56.205673 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:56.205680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:56.205870 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:56.206909 (+ 1039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:56.206977 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:56.209405 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:56.211935 (+ 2530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:56.211942 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:56.211959 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:56.212280 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:56.212701 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:56.212704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:56.212706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:56.212860 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:56.213361 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:56.213366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:56.213642 (+ 276us) client_negotiation.cc:769] Sending connection context
0808 04:34:56.213821 (+ 179us) client_negotiation.cc:240] Negotiation successful
0808 04:34:56.214133 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":178,"threads_started":1}
I20260808 04:34:56.216979 22039 heartbeater.cc:347] Connected to a master server at 127.15.228.126:45977
I20260808 04:34:56.217296 22039 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:56.217901 22039 heartbeater.cc:511] Master 127.15.228.126:45977 requested a full tablet report, sending...
I20260808 04:34:56.219368 21369 ts_manager.cc:194] Registered new tserver with Master: 581c503917544a6b98d9e0f5af3d648d (127.15.228.67:42893)
I20260808 04:34:56.220022 21369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:36185
I20260808 04:34:56.220741 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 04:34:56 dist-test-slave-drl0 krb5kdc[21313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 04:34:56 dist-test-slave-drl0 krb5kdc[21313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.65@KRBTEST.COM
I20260808 04:34:56.256101 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:56.243900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33374 (local address 127.15.228.65:35913)
0808 04:34:56.244238 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:56.244247 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:56.244400 (+ 153us) server_negotiation.cc:407] Connection header received
0808 04:34:56.244485 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:56.244491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:56.244694 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:56.244826 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:56.245902 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:56.247071 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:56.248168 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:56.248687 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:56.248868 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:56.252067 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:56.252080 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:56.252096 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:56.252129 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:56.253992 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:56.254507 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:56.254513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:56.254514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:56.254598 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:56.254892 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:56.254896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:56.254897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:56.255273 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:56.255506 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:56.255719 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 04:34:56.255900 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":142,"threads_started":1}
Aug 08 04:34:56 dist-test-slave-drl0 krb5kdc[21313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:56.270331 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:56.259936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.15.228.126:45977)
0808 04:34:56.260078 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:56.260082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:56.260430 (+ 348us) server_negotiation.cc:407] Connection header received
0808 04:34:56.260528 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:56.260531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:56.260599 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:56.260698 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:56.261454 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:56.262010 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:56.263170 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:56.263478 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:56.263611 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:56.266570 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:56.266574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:56.266577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:56.266623 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:56.268428 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:56.268895 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:56.268898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:56.268900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:56.268989 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:56.269317 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:56.269321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:56.269322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:56.269574 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:56.269695 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:56.269957 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 04:34:56.270119 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 04:34:56.270785 21372 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53362
I20260808 04:34:56.271924 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 21469
I20260808 04:34:56.296984 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 21664
I20260808 04:34:56.317909 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 21865
I20260808 04:34:56.339807 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 21330
2026-08-08T04:34:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2623 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:34:56 dist-test-slave-drl0 krb5kdc[22054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:34:56 dist-test-slave-drl0 krb5kdc[22054](info): set up 2 sockets
Aug 08 04:34:56 dist-test-slave-drl0 krb5kdc[22054](info): commencing operation
krb5kdc: starting...
W20260808 04:34:57.620735 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.004s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:34:57 dist-test-slave-drl0 krb5kdc[22054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:34:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:34:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:34:57.771013 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:33213
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:38717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:33213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:57.904321 22073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:57.904775 22073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:57.904865 22073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:57.910427 22073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:34:57.910591 22073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:57.910633 22073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:57.910676 22073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:34:57.910717 22073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:34:57.916926 22073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:38717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:33213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:33213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:57.918152 22073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:57.919462 22073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:57.931330 22086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:57.931430 22084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:57.931419 22088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:57.931658 22073 server_base.cc:1034] running on GCE node
I20260808 04:34:57.932432 22073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:57.933671 22073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:57.934896 22073 hybrid_clock.cc:648] HybridClock initialized: now 1786163697934882 us; error 30 us; skew 500 ppm
Aug 08 04:34:57 dist-test-slave-drl0 krb5kdc[22054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163697, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:57.938781 22073 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:34:57.939728 22073 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:57.939841 22073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:57.940083 22073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:57.942674 22073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "808b432986a34053846d503f50f7145d"
format_stamp: "Formatted at 2026-08-08 04:34:57 on dist-test-slave-drl0"
server_key: "53f814f2ed3d0681ab51ab84131ef709"
server_key_iv: "8d69059d62ce5fab838efe1476b4c01c"
server_key_version: "encryptionkey@0"
I20260808 04:34:57.943742 22073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "808b432986a34053846d503f50f7145d"
format_stamp: "Formatted at 2026-08-08 04:34:57 on dist-test-slave-drl0"
server_key: "53f814f2ed3d0681ab51ab84131ef709"
server_key_iv: "8d69059d62ce5fab838efe1476b4c01c"
server_key_version: "encryptionkey@0"
I20260808 04:34:57.949846 22073 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 04:34:57.953321 22096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:57.955087 22073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:34:57.955235 22073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "808b432986a34053846d503f50f7145d"
format_stamp: "Formatted at 2026-08-08 04:34:57 on dist-test-slave-drl0"
server_key: "53f814f2ed3d0681ab51ab84131ef709"
server_key_iv: "8d69059d62ce5fab838efe1476b4c01c"
server_key_version: "encryptionkey@0"
I20260808 04:34:57.955413 22073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:57.971719 22073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:57.977105 22073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:57.977376 22073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:57.985926 22073 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:33213
I20260808 04:34:57.986222 22171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:33213 every 8 connection(s)
I20260808 04:34:57.986724 22073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:34:57.990074 22173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:57.992012 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 22073
I20260808 04:34:57.992317 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:34:57.992727 16273 external_mini_cluster.cc:1428] Setting key 79d23ed8c7172cab817b81ae3934dd23
I20260808 04:34:57.997790 22173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d: Bootstrap starting.
I20260808 04:34:58.000380 22173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:58.001408 22173 log.cc:824] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:58.003806 22173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d: No bootstrap required, opened a new log
Aug 08 04:34:58 dist-test-slave-drl0 krb5kdc[22054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:58.007795 22173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808b432986a34053846d503f50f7145d" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33213 } }
I20260808 04:34:58.008018 22173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:58.008052 22173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808b432986a34053846d503f50f7145d, State: Initialized, Role: FOLLOWER
I20260808 04:34:58.008524 22173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808b432986a34053846d503f50f7145d" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33213 } }
I20260808 04:34:58.008649 22173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:34:58.008744 22173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:34:58.008868 22173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:58.009807 22173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808b432986a34053846d503f50f7145d" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33213 } }
I20260808 04:34:58.010215 22173 leader_election.cc:302] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 808b432986a34053846d503f50f7145d; no voters:
I20260808 04:34:58.010593 22173 leader_election.cc:288] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:34:58.010780 22183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:34:58.011039 22183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d [term 1 LEADER]: Becoming Leader. State: Replica: 808b432986a34053846d503f50f7145d, State: Running, Role: LEADER
I20260808 04:34:58.011381 22183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808b432986a34053846d503f50f7145d" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33213 } }
I20260808 04:34:58.011600 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:57.994503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42148 (local address 127.15.228.126:33213)
0808 04:34:57.995107 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:57.995128 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:57.995168 (+ 40us) server_negotiation.cc:407] Connection header received
0808 04:34:57.995947 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:57.995967 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:57.996304 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:57.996657 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:57.997538 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:57.999351 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.000329 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.001273 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.001476 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:58.005684 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:58.005693 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:58.005716 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:58.005771 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:58.007951 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:58.008652 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:58.008660 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:58.008666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:58.008861 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:58.009172 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:58.009175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:58.009177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:58.009606 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:58.009833 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:58.010156 (+ 323us) server_negotiation.cc:299] Negotiation successful
0808 04:34:58.010542 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":154,"threads_started":1}
I20260808 04:34:58.011976 22173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:34:58.012660 22183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 808b432986a34053846d503f50f7145d. Latest consensus state: current_term: 1 leader_uuid: "808b432986a34053846d503f50f7145d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808b432986a34053846d503f50f7145d" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33213 } } }
I20260808 04:34:58.012786 22183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d [sys.catalog]: This master's current role is: LEADER
I20260808 04:34:58.015470 22184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "808b432986a34053846d503f50f7145d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808b432986a34053846d503f50f7145d" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33213 } } }
I20260808 04:34:58.015645 22184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d [sys.catalog]: This master's current role is: LEADER
I20260808 04:34:58.016288 22194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:34:58.019299 22194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:34:58.024996 22194 catalog_manager.cc:1347] Generated new cluster ID: d5c14ccd1c3b45a3b7f141607f1313c8
I20260808 04:34:58.025077 22194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:34:58.032858 22194 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:34:58.033948 22194 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:34:58.044715 22194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d: Generated new TSK 0
I20260808 04:34:58.045393 22194 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
I20260808 04:34:58.087574 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33213
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:38717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:58.228988 22212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:58.229358 22212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:58.229426 22212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:58.233567 22212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:58.233635 22212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:58.233698 22212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:34:58.239399 22212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:38717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:58.240509 22212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:58.241590 22212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:58.254019 22229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:58.254019 22227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:58.254110 22225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:58.254496 22212 server_base.cc:1034] running on GCE node
I20260808 04:34:58.255199 22212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:58.256135 22212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:58.257378 22212 hybrid_clock.cc:648] HybridClock initialized: now 1786163698257332 us; error 56 us; skew 500 ppm
Aug 08 04:34:58 dist-test-slave-drl0 krb5kdc[22054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163698, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:58.261026 22212 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:34:58.262261 22212 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:58.262391 22212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:58.262722 22212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:58.265731 22212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "d67261c4f1bd427693b6a4d67c626042"
format_stamp: "Formatted at 2026-08-08 04:34:58 on dist-test-slave-drl0"
server_key: "43e2c6e40f0f65c2aeb2d1b071b3a503"
server_key_iv: "6bdc927876132c7607e35c3944b010e7"
server_key_version: "encryptionkey@0"
I20260808 04:34:58.267032 22212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "d67261c4f1bd427693b6a4d67c626042"
format_stamp: "Formatted at 2026-08-08 04:34:58 on dist-test-slave-drl0"
server_key: "43e2c6e40f0f65c2aeb2d1b071b3a503"
server_key_iv: "6bdc927876132c7607e35c3944b010e7"
server_key_version: "encryptionkey@0"
I20260808 04:34:58.271960 22212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:34:58.275450 22238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:58.277047 22212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:34:58.277211 22212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "d67261c4f1bd427693b6a4d67c626042"
format_stamp: "Formatted at 2026-08-08 04:34:58 on dist-test-slave-drl0"
server_key: "43e2c6e40f0f65c2aeb2d1b071b3a503"
server_key_iv: "6bdc927876132c7607e35c3944b010e7"
server_key_version: "encryptionkey@0"
I20260808 04:34:58.277355 22212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:58.303902 22212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:58.307878 22212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:58.308319 22212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:58.309005 22212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:34:58.309908 22212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:58.309980 22212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:58.310051 22212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:58.310096 22212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:58.321102 22212 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:40907
I20260808 04:34:58.321169 22389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:40907 every 8 connection(s)
I20260808 04:34:58.321960 22212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:34:58.328135 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 22212
I20260808 04:34:58.328395 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:34:58.328974 16273 external_mini_cluster.cc:1428] Setting key 69c8ecce25254fe88498fb9a5b998f29
Aug 08 04:34:58 dist-test-slave-drl0 krb5kdc[22054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163698, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:58.341040 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:58.324502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:36331 (local address 127.15.228.126:33213)
0808 04:34:58.324790 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:58.324794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:58.326369 (+ 1575us) server_negotiation.cc:407] Connection header received
0808 04:34:58.327801 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:58.327806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:58.327894 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:58.327997 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:58.329600 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.330279 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.331849 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.332247 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.332393 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:58.335462 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:58.335469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:58.335474 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:58.335509 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:58.337643 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:58.338339 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:58.338345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:58.338349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:58.338455 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:58.339364 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:58.339370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:58.339372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:58.339644 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:58.339781 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:58.340604 (+ 823us) server_negotiation.cc:299] Negotiation successful
0808 04:34:58.340776 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
WARNING: no policy specified for kudu/127.15.228.66@KRBTEST.COM; defaulting to no policy
I20260808 04:34:58.342501 22393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:58.325049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:33213 (local address 127.15.228.65:36331)
0808 04:34:58.326142 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:58.326193 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:58.327487 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:58.328214 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:58.328221 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:58.328821 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:58.329433 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:58.329448 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.330486 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.330493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:58.331613 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:58.331624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.332467 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.332470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:58.332626 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:58.333361 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:58.333379 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:58.335278 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:58.337819 (+ 2541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:58.337825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:58.337839 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:58.338126 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:58.338691 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:58.338696 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:58.338698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:58.339169 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:58.339813 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:58.339821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:58.340256 (+ 435us) client_negotiation.cc:769] Sending connection context
0808 04:34:58.340482 (+ 226us) client_negotiation.cc:240] Negotiation successful
0808 04:34:58.340906 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":159,"threads_started":1}
I20260808 04:34:58.344506 22390 heartbeater.cc:347] Connected to a master server at 127.15.228.126:33213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.66@KRBTEST.COM" created.
I20260808 04:34:58.345125 22390 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:58.346155 22390 heartbeater.cc:511] Master 127.15.228.126:33213 requested a full tablet report, sending...
I20260808 04:34:58.348511 22120 ts_manager.cc:194] Registered new tserver with Master: d67261c4f1bd427693b6a4d67c626042 (127.15.228.65:40907)
I20260808 04:34:58.349901 22120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:36331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
I20260808 04:34:58.367830 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33213
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:38717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:58.504631 22399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:58.504935 22399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:58.504988 22399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:58.509724 22399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:58.509797 22399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:58.509861 22399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:34:58.516630 22399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:38717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:58.517814 22399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:58.519403 22399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:58.531311 22408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:58.531469 22406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:58.531641 22410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:58.531840 22399 server_base.cc:1034] running on GCE node
I20260808 04:34:58.532527 22399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:58.533402 22399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:58.534626 22399 hybrid_clock.cc:648] HybridClock initialized: now 1786163698534591 us; error 48 us; skew 500 ppm
Aug 08 04:34:58 dist-test-slave-drl0 krb5kdc[22054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163698, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:58.538380 22399 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:34:58.539318 22399 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:58.539392 22399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:58.539606 22399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:58.542377 22399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "a6fd9cf0ba4340f4aca8307d1bd1b532"
format_stamp: "Formatted at 2026-08-08 04:34:58 on dist-test-slave-drl0"
server_key: "a519961790d9f36a7bb61e721942398c"
server_key_iv: "01e9c73e11ce1233729f42920094c0b0"
server_key_version: "encryptionkey@0"
I20260808 04:34:58.543102 22399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "a6fd9cf0ba4340f4aca8307d1bd1b532"
format_stamp: "Formatted at 2026-08-08 04:34:58 on dist-test-slave-drl0"
server_key: "a519961790d9f36a7bb61e721942398c"
server_key_iv: "01e9c73e11ce1233729f42920094c0b0"
server_key_version: "encryptionkey@0"
I20260808 04:34:58.548763 22399 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:34:58.551753 22416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:58.553275 22399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:34:58.553444 22399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "a6fd9cf0ba4340f4aca8307d1bd1b532"
format_stamp: "Formatted at 2026-08-08 04:34:58 on dist-test-slave-drl0"
server_key: "a519961790d9f36a7bb61e721942398c"
server_key_iv: "01e9c73e11ce1233729f42920094c0b0"
server_key_version: "encryptionkey@0"
I20260808 04:34:58.553572 22399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:58.585451 22399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:58.590260 22399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:58.590517 22399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:58.591289 22399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:34:58.592227 22399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:58.592311 22399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:58.592383 22399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:58.592438 22399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:58.606232 22399 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:37997
I20260808 04:34:58.606261 22599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:37997 every 8 connection(s)
I20260808 04:34:58.607157 22399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:34:58.607493 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 22399
I20260808 04:34:58.607631 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:34:58.608011 16273 external_mini_cluster.cc:1428] Setting key 8f33bc3dbaf3d940519c3458336813a6
WARNING: no policy specified for kudu/127.15.228.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.67@KRBTEST.COM" created.
Aug 08 04:34:58 dist-test-slave-drl0 krb5kdc[22054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163698, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:58.630472 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:58.609608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:49411 (local address 127.15.228.126:33213)
0808 04:34:58.609785 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:58.609790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:58.611444 (+ 1654us) server_negotiation.cc:407] Connection header received
0808 04:34:58.612875 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:58.612881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:58.613027 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:58.613110 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:58.614372 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.615441 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.617677 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.617908 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.618337 (+ 429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:58.625071 (+ 6734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:58.625077 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:58.625084 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:58.625129 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:58.627367 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:58.628183 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:58.628186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:58.628187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:58.628267 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:58.628779 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:58.628783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:58.628785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:58.629052 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:58.629150 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:58.630077 (+ 927us) server_negotiation.cc:299] Negotiation successful
0808 04:34:58.630293 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:34:58.631608 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:58.610193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:33213 (local address 127.15.228.66:49411)
0808 04:34:58.611268 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:58.611328 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:58.612564 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:58.613328 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:58.613335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:58.613700 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:58.614211 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:58.614224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.615661 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.615667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:58.617475 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:58.617489 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.618348 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.618356 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:58.618603 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:58.619416 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:58.619442 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:58.624775 (+ 5333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:58.627533 (+ 2758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:58.627544 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:58.627567 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:58.628035 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:58.628379 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:58.628386 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:58.628391 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:58.628608 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:58.629184 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:58.629195 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:58.629606 (+ 411us) client_negotiation.cc:769] Sending connection context
0808 04:34:58.629882 (+ 276us) client_negotiation.cc:240] Negotiation successful
0808 04:34:58.630340 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":222,"threads_started":1}
I20260808 04:34:58.633373 22601 heartbeater.cc:347] Connected to a master server at 127.15.228.126:33213
I20260808 04:34:58.633654 22601 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:58.634155 22601 heartbeater.cc:511] Master 127.15.228.126:33213 requested a full tablet report, sending...
I20260808 04:34:58.635847 22120 ts_manager.cc:194] Registered new tserver with Master: a6fd9cf0ba4340f4aca8307d1bd1b532 (127.15.228.66:37997)
I20260808 04:34:58.636606 22120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:49411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
I20260808 04:34:58.643301 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33213
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:38717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:34:58.774127 22607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:58.774452 22607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:58.774536 22607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:58.779157 22607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:58.779287 22607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:58.779418 22607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:34:58.786473 22607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:38717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:58.788228 22607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:58.789790 22607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:58.801927 22616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:58.801954 22620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:58.802299 22617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:58.803231 22607 server_base.cc:1034] running on GCE node
I20260808 04:34:58.803731 22607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:58.804419 22607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:58.805714 22607 hybrid_clock.cc:648] HybridClock initialized: now 1786163698805686 us; error 47 us; skew 500 ppm
Aug 08 04:34:58 dist-test-slave-drl0 krb5kdc[22054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163698, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:34:58.810303 22607 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:34:58.811582 22607 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:58.811685 22607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:58.811949 22607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:58.814785 22607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "31fb97ade29b4f7e851a321c4012fad5"
format_stamp: "Formatted at 2026-08-08 04:34:58 on dist-test-slave-drl0"
server_key: "c4e0180b395c4cf69097caa570acbc0a"
server_key_iv: "1244e6dfe5b71018c2979939efba0074"
server_key_version: "encryptionkey@0"
I20260808 04:34:58.815789 22607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "31fb97ade29b4f7e851a321c4012fad5"
format_stamp: "Formatted at 2026-08-08 04:34:58 on dist-test-slave-drl0"
server_key: "c4e0180b395c4cf69097caa570acbc0a"
server_key_iv: "1244e6dfe5b71018c2979939efba0074"
server_key_version: "encryptionkey@0"
I20260808 04:34:58.821627 22607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:34:58.825572 22629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:58.827962 22607 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260808 04:34:58.828256 22607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "31fb97ade29b4f7e851a321c4012fad5"
format_stamp: "Formatted at 2026-08-08 04:34:58 on dist-test-slave-drl0"
server_key: "c4e0180b395c4cf69097caa570acbc0a"
server_key_iv: "1244e6dfe5b71018c2979939efba0074"
server_key_version: "encryptionkey@0"
I20260808 04:34:58.828421 22607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:58.843537 22607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:58.847728 22607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:58.847972 22607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:58.848639 22607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:34:58.849545 22607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:58.849617 22607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:58.849697 22607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:58.849751 22607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:58.863940 22607 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:37229
I20260808 04:34:58.863960 22791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:37229 every 8 connection(s)
I20260808 04:34:58.864661 22607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:34:58.872660 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 22607
I20260808 04:34:58.872856 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:34:58.873248 16273 external_mini_cluster.cc:1428] Setting key eeca3221137666dcbabde08f5a869620
Aug 08 04:34:58 dist-test-slave-drl0 krb5kdc[22054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163698, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:34:58.883003 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:58.867099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:38555 (local address 127.15.228.126:33213)
0808 04:34:58.867342 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:58.867346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:58.867967 (+ 621us) server_negotiation.cc:407] Connection header received
0808 04:34:58.869504 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:58.869507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:58.869571 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:58.869651 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:58.871371 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.872024 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.874051 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.874354 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.874474 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:58.878327 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:58.878331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:58.878333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:58.878365 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:58.879920 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:58.880568 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:58.880571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:58.880573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:58.880654 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:58.881089 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:58.881100 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:58.881101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:58.881336 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:58.881418 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:58.882281 (+ 863us) server_negotiation.cc:299] Negotiation successful
0808 04:34:58.882577 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:34:58.883463 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:58.867253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:33213 (local address 127.15.228.67:38555)
0808 04:34:58.867804 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:58.867848 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:58.869252 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:58.869822 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:58.869828 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:58.870228 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:58.871058 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:58.871120 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.872207 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.872252 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:58.873876 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:58.873890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.874804 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.874817 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:58.875117 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:58.875864 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:34:58.875882 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:58.877939 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:58.880075 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:58.880082 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:58.880097 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:58.880411 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:58.880767 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:34:58.880770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:34:58.880772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:34:58.880938 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:34:58.881470 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:58.881477 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:58.881863 (+ 386us) client_negotiation.cc:769] Sending connection context
0808 04:34:58.882043 (+ 180us) client_negotiation.cc:240] Negotiation successful
0808 04:34:58.882356 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":118,"threads_started":1}
I20260808 04:34:58.885376 22794 heartbeater.cc:347] Connected to a master server at 127.15.228.126:33213
I20260808 04:34:58.885728 22794 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:58.886397 22794 heartbeater.cc:511] Master 127.15.228.126:33213 requested a full tablet report, sending...
I20260808 04:34:58.888068 22120 ts_manager.cc:194] Registered new tserver with Master: 31fb97ade29b4f7e851a321c4012fad5 (127.15.228.67:37229)
I20260808 04:34:58.888737 22120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:38555
I20260808 04:34:58.889582 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:34:58.900276 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:58.891945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42156 (local address 127.15.228.126:33213)
0808 04:34:58.892106 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:58.892111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:58.892234 (+ 123us) server_negotiation.cc:407] Connection header received
0808 04:34:58.892401 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:58.892404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:58.892452 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:58.892548 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:58.893290 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.893817 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.894656 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.894892 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.895018 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:58.896579 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:58.896584 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:58.896586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:34:58.896611 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:58.898434 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:58.898885 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:58.898889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:58.898890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:58.899038 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:34:58.899324 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:34:58.899328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:34:58.899330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:34:58.899565 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:58.899711 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:58.899955 (+ 244us) server_negotiation.cc:299] Negotiation successful
0808 04:34:58.900076 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 04:34:58.905085 22120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:34:58.909409 22120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:34:58.926862 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:58.920378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52840 (local address 127.15.228.66:37997)
0808 04:34:58.921258 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:58.921270 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:58.921294 (+ 24us) server_negotiation.cc:407] Connection header received
0808 04:34:58.921375 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:58.921380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:58.921593 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:58.922195 (+ 602us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:58.922350 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.923561 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.925529 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.926107 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.926386 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:58.926525 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:58.926580 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 04:34:58.926672 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":135,"threads_started":1}
I20260808 04:34:58.926946 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:58.919998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44396 (local address 127.15.228.65:40907)
0808 04:34:58.921339 (+ 1341us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:58.921347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:58.921374 (+ 27us) server_negotiation.cc:407] Connection header received
0808 04:34:58.921464 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:58.921470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:58.921675 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:58.921876 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:58.922165 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.923430 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.925553 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.926329 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.926473 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:58.926603 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:58.926692 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 04:34:58.926797 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1224,"thread_start_us":108,"threads_started":1}
I20260808 04:34:58.926919 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:58.919840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:40907 (local address 127.0.0.1:44396)
0808 04:34:58.920770 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:58.920835 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:58.921013 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:58.921851 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:58.921853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:58.921865 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:58.922031 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:58.922039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.923579 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.923586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:58.925416 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:58.925426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.926479 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.926483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:58.926569 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:58.926592 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:34:58.926664 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 04:34:58.926748 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":341,"spinlock_wait_cycles":24064,"thread_start_us":114,"threads_started":1}
I20260808 04:34:58.927139 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:58.919828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:37997 (local address 127.0.0.1:52840)
0808 04:34:58.920770 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:58.920835 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:58.921013 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:58.921803 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:58.921809 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:58.921836 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:58.922031 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:58.922039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.923723 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.923727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:58.925379 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:58.925391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.926275 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.926278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:58.926375 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:58.926425 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 04:34:58.926534 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 04:34:58.926661 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":216,"thread_start_us":139,"threads_started":1}
I20260808 04:34:58.927508 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:58.920604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49866 (local address 127.15.228.67:37229)
0808 04:34:58.920994 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:58.921007 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:58.921431 (+ 424us) server_negotiation.cc:407] Connection header received
0808 04:34:58.921542 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:58.921547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:58.921709 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:58.921876 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:58.922392 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.923561 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.925218 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.926060 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.926177 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:58.926371 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:58.927208 (+ 837us) server_negotiation.cc:299] Negotiation successful
0808 04:34:58.927308 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":184,"threads_started":1}
I20260808 04:34:58.930621 22295 tablet_service.cc:1467] Processing CreateTablet for tablet f402272ddd2e48ed88c6c837525a7890 (DEFAULT_TABLE table=test-table [id=6f975903ad1e467f8c309b8aa4d0aeff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:58.930621 22491 tablet_service.cc:1467] Processing CreateTablet for tablet f402272ddd2e48ed88c6c837525a7890 (DEFAULT_TABLE table=test-table [id=6f975903ad1e467f8c309b8aa4d0aeff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:58.931965 22491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f402272ddd2e48ed88c6c837525a7890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:58.931965 22295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f402272ddd2e48ed88c6c837525a7890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:58.933284 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:58.919747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:37229 (local address 127.0.0.1:49866)
0808 04:34:58.921225 (+ 1478us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:58.921239 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:58.921328 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:58.922056 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:58.922058 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:58.922077 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:58.922245 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:58.922254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.923732 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.923735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:58.925088 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:58.925098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.926204 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.926210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:58.926791 (+ 581us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:58.926814 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:34:58.926892 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 04:34:58.933133 (+ 6241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":187,"thread_start_us":86,"threads_started":1}
I20260808 04:34:58.936638 22694 tablet_service.cc:1467] Processing CreateTablet for tablet f402272ddd2e48ed88c6c837525a7890 (DEFAULT_TABLE table=test-table [id=6f975903ad1e467f8c309b8aa4d0aeff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:34:58.938561 22694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f402272ddd2e48ed88c6c837525a7890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:58.941104 22819 tablet_bootstrap.cc:492] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042: Bootstrap starting.
I20260808 04:34:58.943888 22819 tablet_bootstrap.cc:654] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:58.944870 22819 log.cc:824] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:58.946640 22821 tablet_bootstrap.cc:492] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5: Bootstrap starting.
I20260808 04:34:58.947501 22819 tablet_bootstrap.cc:492] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042: No bootstrap required, opened a new log
I20260808 04:34:58.947729 22819 ts_tablet_manager.cc:1397] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260808 04:34:58.948526 22822 tablet_bootstrap.cc:492] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532: Bootstrap starting.
I20260808 04:34:58.949299 22821 tablet_bootstrap.cc:654] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:58.950513 22821 log.cc:824] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:58.951210 22822 tablet_bootstrap.cc:654] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:58.952350 22822 log.cc:824] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:58.952817 22819 raft_consensus.cc:348] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6fd9cf0ba4340f4aca8307d1bd1b532" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 37997 } } peers { permanent_uuid: "d67261c4f1bd427693b6a4d67c626042" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40907 } } peers { permanent_uuid: "31fb97ade29b4f7e851a321c4012fad5" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37229 } }
I20260808 04:34:58.953039 22819 raft_consensus.cc:374] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:58.953135 22819 raft_consensus.cc:729] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67261c4f1bd427693b6a4d67c626042, State: Initialized, Role: FOLLOWER
I20260808 04:34:58.953197 22821 tablet_bootstrap.cc:492] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5: No bootstrap required, opened a new log
I20260808 04:34:58.953408 22821 ts_tablet_manager.cc:1397] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:34:58.953686 22819 consensus_queue.cc:260] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6fd9cf0ba4340f4aca8307d1bd1b532" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 37997 } } peers { permanent_uuid: "d67261c4f1bd427693b6a4d67c626042" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40907 } } peers { permanent_uuid: "31fb97ade29b4f7e851a321c4012fad5" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37229 } }
I20260808 04:34:58.954433 22390 heartbeater.cc:503] Master 127.15.228.126:33213 was elected leader, sending a full tablet report...
I20260808 04:34:58.954734 22819 ts_tablet_manager.cc:1428] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 04:34:58.955475 22822 tablet_bootstrap.cc:492] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532: No bootstrap required, opened a new log
I20260808 04:34:58.955695 22822 ts_tablet_manager.cc:1397] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:34:58.957365 22821 raft_consensus.cc:348] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6fd9cf0ba4340f4aca8307d1bd1b532" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 37997 } } peers { permanent_uuid: "d67261c4f1bd427693b6a4d67c626042" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40907 } } peers { permanent_uuid: "31fb97ade29b4f7e851a321c4012fad5" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37229 } }
I20260808 04:34:58.957754 22821 raft_consensus.cc:374] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:58.957808 22821 raft_consensus.cc:729] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31fb97ade29b4f7e851a321c4012fad5, State: Initialized, Role: FOLLOWER
I20260808 04:34:58.958884 22821 consensus_queue.cc:260] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6fd9cf0ba4340f4aca8307d1bd1b532" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 37997 } } peers { permanent_uuid: "d67261c4f1bd427693b6a4d67c626042" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40907 } } peers { permanent_uuid: "31fb97ade29b4f7e851a321c4012fad5" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37229 } }
I20260808 04:34:58.960583 22822 raft_consensus.cc:348] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6fd9cf0ba4340f4aca8307d1bd1b532" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 37997 } } peers { permanent_uuid: "d67261c4f1bd427693b6a4d67c626042" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40907 } } peers { permanent_uuid: "31fb97ade29b4f7e851a321c4012fad5" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37229 } }
I20260808 04:34:58.960896 22822 raft_consensus.cc:374] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:58.960968 22822 raft_consensus.cc:729] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6fd9cf0ba4340f4aca8307d1bd1b532, State: Initialized, Role: FOLLOWER
I20260808 04:34:58.961367 22821 ts_tablet_manager.cc:1428] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5: Time spent starting tablet: real 0.008s user 0.001s sys 0.006s
I20260808 04:34:58.961414 22822 consensus_queue.cc:260] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6fd9cf0ba4340f4aca8307d1bd1b532" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 37997 } } peers { permanent_uuid: "d67261c4f1bd427693b6a4d67c626042" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40907 } } peers { permanent_uuid: "31fb97ade29b4f7e851a321c4012fad5" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37229 } }
I20260808 04:34:58.961771 22794 heartbeater.cc:503] Master 127.15.228.126:33213 was elected leader, sending a full tablet report...
I20260808 04:34:58.962147 22601 heartbeater.cc:503] Master 127.15.228.126:33213 was elected leader, sending a full tablet report...
I20260808 04:34:58.962387 22822 ts_tablet_manager.cc:1428] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 04:34:58.962821 22826 raft_consensus.cc:482] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:34:58.963114 22826 raft_consensus.cc:504] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6fd9cf0ba4340f4aca8307d1bd1b532" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 37997 } } peers { permanent_uuid: "d67261c4f1bd427693b6a4d67c626042" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40907 } } peers { permanent_uuid: "31fb97ade29b4f7e851a321c4012fad5" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37229 } }
I20260808 04:34:58.964191 22826 leader_election.cc:288] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6fd9cf0ba4340f4aca8307d1bd1b532 (127.15.228.66:37997), 31fb97ade29b4f7e851a321c4012fad5 (127.15.228.67:37229)
I20260808 04:34:58.969383 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:58.964529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:47221 (local address 127.15.228.67:37229)
0808 04:34:58.964692 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:58.964699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:58.964984 (+ 285us) server_negotiation.cc:407] Connection header received
0808 04:34:58.965331 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:58.965334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:58.965381 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:58.965470 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:58.965852 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.966732 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.968124 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.968770 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.968904 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:58.968961 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:58.969215 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 04:34:58.969279 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:34:58.969466 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:58.964453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:37229 (local address 127.15.228.65:47221)
0808 04:34:58.964890 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:58.964908 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:58.965196 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:58.965501 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:58.965504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:58.965523 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:58.965700 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:58.965709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.966873 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.966876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:58.967993 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:58.968003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.968924 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.968927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:58.969065 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:58.969097 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 04:34:58.969200 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 04:34:58.969278 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":20,"thread_start_us":116,"threads_started":1}
I20260808 04:34:58.970137 22724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f402272ddd2e48ed88c6c837525a7890" candidate_uuid: "d67261c4f1bd427693b6a4d67c626042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31fb97ade29b4f7e851a321c4012fad5" is_pre_election: true
I20260808 04:34:58.970541 22724 raft_consensus.cc:2454] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d67261c4f1bd427693b6a4d67c626042 in term 0.
I20260808 04:34:58.970795 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:58.964540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:37997 (local address 127.15.228.65:50445)
0808 04:34:58.965046 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:58.965060 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:58.966098 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:58.966680 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:58.966683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:58.966704 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:34:58.966828 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:58.966838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.967747 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.967752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:58.968927 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:34:58.968937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.970039 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.970043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:34:58.970207 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:58.970244 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 04:34:58.970436 (+ 192us) client_negotiation.cc:240] Negotiation successful
0808 04:34:58.970554 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":19,"thread_start_us":110,"threads_started":1}
I20260808 04:34:58.970860 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:58.964963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:50445 (local address 127.15.228.66:37997)
0808 04:34:58.965105 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:58.965110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:58.965124 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:34:58.966317 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:58.966320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:58.966370 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:58.966420 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:34:58.966961 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.967629 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.969072 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.969799 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.969950 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:58.970000 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:58.970467 (+ 467us) server_negotiation.cc:299] Negotiation successful
0808 04:34:58.970556 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:34:58.971493 22242 leader_election.cc:302] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31fb97ade29b4f7e851a321c4012fad5, d67261c4f1bd427693b6a4d67c626042; no voters:
I20260808 04:34:58.971616 22522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f402272ddd2e48ed88c6c837525a7890" candidate_uuid: "d67261c4f1bd427693b6a4d67c626042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6fd9cf0ba4340f4aca8307d1bd1b532" is_pre_election: true
I20260808 04:34:58.971885 22826 raft_consensus.cc:2781] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:34:58.971909 22522 raft_consensus.cc:2454] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d67261c4f1bd427693b6a4d67c626042 in term 0.
I20260808 04:34:58.971972 22826 raft_consensus.cc:482] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:34:58.972018 22826 raft_consensus.cc:3037] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:58.973335 22826 raft_consensus.cc:504] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6fd9cf0ba4340f4aca8307d1bd1b532" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 37997 } } peers { permanent_uuid: "d67261c4f1bd427693b6a4d67c626042" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40907 } } peers { permanent_uuid: "31fb97ade29b4f7e851a321c4012fad5" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37229 } }
I20260808 04:34:58.973704 22826 leader_election.cc:288] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [CANDIDATE]: Term 1 election: Requested vote from peers a6fd9cf0ba4340f4aca8307d1bd1b532 (127.15.228.66:37997), 31fb97ade29b4f7e851a321c4012fad5 (127.15.228.67:37229)
I20260808 04:34:58.974228 22522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f402272ddd2e48ed88c6c837525a7890" candidate_uuid: "d67261c4f1bd427693b6a4d67c626042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6fd9cf0ba4340f4aca8307d1bd1b532"
I20260808 04:34:58.974360 22522 raft_consensus.cc:3037] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:58.974596 22724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f402272ddd2e48ed88c6c837525a7890" candidate_uuid: "d67261c4f1bd427693b6a4d67c626042" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31fb97ade29b4f7e851a321c4012fad5"
I20260808 04:34:58.974828 22724 raft_consensus.cc:3037] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:58.975616 22522 raft_consensus.cc:2454] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d67261c4f1bd427693b6a4d67c626042 in term 1.
I20260808 04:34:58.976040 22240 leader_election.cc:302] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6fd9cf0ba4340f4aca8307d1bd1b532, d67261c4f1bd427693b6a4d67c626042; no voters:
I20260808 04:34:58.976212 22826 raft_consensus.cc:2781] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:34:58.976454 22826 raft_consensus.cc:686] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [term 1 LEADER]: Becoming Leader. State: Replica: d67261c4f1bd427693b6a4d67c626042, State: Running, Role: LEADER
I20260808 04:34:58.976799 22724 raft_consensus.cc:2454] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d67261c4f1bd427693b6a4d67c626042 in term 1.
I20260808 04:34:58.976806 22826 consensus_queue.cc:237] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6fd9cf0ba4340f4aca8307d1bd1b532" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 37997 } } peers { permanent_uuid: "d67261c4f1bd427693b6a4d67c626042" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40907 } } peers { permanent_uuid: "31fb97ade29b4f7e851a321c4012fad5" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37229 } }
I20260808 04:34:58.980397 22115 catalog_manager.cc:5495] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 reported cstate change: term changed from 0 to 1, leader changed from <none> to d67261c4f1bd427693b6a4d67c626042 (127.15.228.65). New cstate: current_term: 1 leader_uuid: "d67261c4f1bd427693b6a4d67c626042" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6fd9cf0ba4340f4aca8307d1bd1b532" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 37997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d67261c4f1bd427693b6a4d67c626042" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31fb97ade29b4f7e851a321c4012fad5" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37229 } health_report { overall_health: UNKNOWN } } }
I20260808 04:34:58.996088 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:58.991422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44402 (local address 127.15.228.65:40907)
0808 04:34:58.991554 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:58.991559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:58.991572 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:34:58.991701 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:58.991704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:58.991767 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:58.991909 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:34:58.992293 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.993122 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.994163 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:34:58.994507 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:34:58.994619 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:34:58.995003 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:34:58.995508 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:34:58.995627 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:58.995751 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 04:34:58.995811 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 04:34:59.003569 22724 raft_consensus.cc:1261] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5 [term 1 FOLLOWER]: Refusing update from remote peer d67261c4f1bd427693b6a4d67c626042: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:34:59.003611 22522 raft_consensus.cc:1261] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532 [term 1 FOLLOWER]: Refusing update from remote peer d67261c4f1bd427693b6a4d67c626042: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:34:59.004616 22826 consensus_queue.cc:1035] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31fb97ade29b4f7e851a321c4012fad5" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:34:59.004925 22833 consensus_queue.cc:1035] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6fd9cf0ba4340f4aca8307d1bd1b532" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 37997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:34:59.018294 22838 mvcc.cc:205] Tried to move back new op lower bound from 7316126511106793472 to 7316126511012286464. Current Snapshot: MvccSnapshot[applied={T|T < 7316126511106793472 or (T in {7316126511106793472})}]
I20260808 04:34:59.023619 22120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:34:59.023873 22120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:34:59.027748 22120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 808b432986a34053846d503f50f7145d: Sending DeleteTablet for 3 replicas of tablet f402272ddd2e48ed88c6c837525a7890
I20260808 04:34:59.028854 22295 tablet_service.cc:1514] Processing DeleteTablet for tablet f402272ddd2e48ed88c6c837525a7890 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:59 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:44396
I20260808 04:34:59.028854 22491 tablet_service.cc:1514] Processing DeleteTablet for tablet f402272ddd2e48ed88c6c837525a7890 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:59 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:52840
I20260808 04:34:59.029407 22850 tablet_replica.cc:330] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042: stopping tablet replica
I20260808 04:34:59.029348 22694 tablet_service.cc:1514] Processing DeleteTablet for tablet f402272ddd2e48ed88c6c837525a7890 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:34:59 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:49866
I20260808 04:34:59.029630 22853 tablet_replica.cc:330] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532: stopping tablet replica
I20260808 04:34:59.029772 22850 raft_consensus.cc:2229] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:34:59.030046 22853 raft_consensus.cc:2229] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:34:59.030156 22850 raft_consensus.cc:2258] T f402272ddd2e48ed88c6c837525a7890 P d67261c4f1bd427693b6a4d67c626042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:34:59.030465 22853 raft_consensus.cc:2258] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:34:59.030793 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 22212
I20260808 04:34:59.038112 22853 ts_tablet_manager.cc:1905] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:34:59.041638 22853 ts_tablet_manager.cc:1918] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:34:59.041848 22853 log.cc:1196] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/f402272ddd2e48ed88c6c837525a7890
I20260808 04:34:59.042236 22853 ts_tablet_manager.cc:1939] T f402272ddd2e48ed88c6c837525a7890 P a6fd9cf0ba4340f4aca8307d1bd1b532: Deleting consensus metadata
I20260808 04:34:59.042542 22854 tablet_replica.cc:330] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5: stopping tablet replica
I20260808 04:34:59.044082 22098 catalog_manager.cc:4841] TS a6fd9cf0ba4340f4aca8307d1bd1b532 (127.15.228.66:37997): tablet f402272ddd2e48ed88c6c837525a7890 (table test-table [id=6f975903ad1e467f8c309b8aa4d0aeff]) successfully deleted
I20260808 04:34:59.044672 22854 raft_consensus.cc:2229] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:34:59.045068 22854 raft_consensus.cc:2258] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:34:59.047474 22854 ts_tablet_manager.cc:1905] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:34:59.050305 22854 ts_tablet_manager.cc:1918] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:34:59.050434 22854 log.cc:1196] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/f402272ddd2e48ed88c6c837525a7890
I20260808 04:34:59.050818 22854 ts_tablet_manager.cc:1939] T f402272ddd2e48ed88c6c837525a7890 P 31fb97ade29b4f7e851a321c4012fad5: Deleting consensus metadata
I20260808 04:34:59.052367 22103 catalog_manager.cc:4841] TS 31fb97ade29b4f7e851a321c4012fad5 (127.15.228.67:37229): tablet f402272ddd2e48ed88c6c837525a7890 (table test-table [id=6f975903ad1e467f8c309b8aa4d0aeff]) successfully deleted
W20260808 04:34:59.058552 22101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.228.65:40907 (error 108)
I20260808 04:34:59.058724 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 22399
I20260808 04:34:59.063320 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:59.062898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:40907 (local address 127.0.0.1:44408)
0808 04:34:59.063130 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:59.063208 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.65:40907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260808 04:34:59.063652 22101 catalog_manager.cc:4584] TS d67261c4f1bd427693b6a4d67c626042 (127.15.228.65:40907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f402272ddd2e48ed88c6c837525a7890: Network error: Client connection negotiation failed: client connection to 127.15.228.65:40907: connect: Connection refused (error 111)
I20260808 04:34:59.071316 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 22607
I20260808 04:34:59.084520 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 22073
2026-08-08T04:34:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2744 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T04:34:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:34:59Z Disabled control of system clock
I20260808 04:34:59.125710 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:39741
--webserver_interface=127.15.228.126
--webserver_port=0
--builtin_ntp_servers=127.15.228.84:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:39741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 04:34:59.249635 22860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:59.249925 22860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:59.249964 22860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:59.254776 22860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:34:59.254916 22860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:59.255061 22860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:59.255105 22860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:34:59.255164 22860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:34:59.262509 22860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:39741
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:39741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:59.264209 22860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:59.266377 22860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:59.275663 22872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:59.275666 22866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:59.275663 22867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:59.276137 22860 server_base.cc:1034] running on GCE node
I20260808 04:34:59.276743 22860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:59.277913 22860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:59.279136 22860 hybrid_clock.cc:648] HybridClock initialized: now 1786163699279092 us; error 38 us; skew 500 ppm
I20260808 04:34:59.281822 22860 webserver.cc:460] Webserver started at http://127.15.228.126:41963/ using document root <none> and password file <none>
I20260808 04:34:59.282495 22860 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:59.282593 22860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:59.282837 22860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:59.285400 22860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "0c0ff66ee23e4ab09faede8f0b9b965b"
format_stamp: "Formatted at 2026-08-08 04:34:59 on dist-test-slave-drl0"
server_key: "7da200e5ddb2103ae8e9e2e61f7759c5"
server_key_iv: "a9c546a9a873fb80b5bf7a593c1073c1"
server_key_version: "encryptionkey@0"
I20260808 04:34:59.286016 22860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "0c0ff66ee23e4ab09faede8f0b9b965b"
format_stamp: "Formatted at 2026-08-08 04:34:59 on dist-test-slave-drl0"
server_key: "7da200e5ddb2103ae8e9e2e61f7759c5"
server_key_iv: "a9c546a9a873fb80b5bf7a593c1073c1"
server_key_version: "encryptionkey@0"
I20260808 04:34:59.289925 22860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:34:59.292768 22881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:59.294235 22860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:34:59.294397 22860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "0c0ff66ee23e4ab09faede8f0b9b965b"
format_stamp: "Formatted at 2026-08-08 04:34:59 on dist-test-slave-drl0"
server_key: "7da200e5ddb2103ae8e9e2e61f7759c5"
server_key_iv: "a9c546a9a873fb80b5bf7a593c1073c1"
server_key_version: "encryptionkey@0"
I20260808 04:34:59.294519 22860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:59.299326 22860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:59.299957 22860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:59.300141 22860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:59.310268 22860 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:39741
I20260808 04:34:59.310359 22955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:39741 every 8 connection(s)
I20260808 04:34:59.311374 22860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:34:59.312830 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 22860
I20260808 04:34:59.312928 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:34:59.313346 16273 external_mini_cluster.cc:1428] Setting key 57882acff7983a10c2c3c8cc355d73ef
I20260808 04:34:59.315397 22958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:34:59.320109 22963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:59.314759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46280 (local address 127.15.228.126:39741)
0808 04:34:59.315613 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:59.315634 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:59.315679 (+ 45us) server_negotiation.cc:407] Connection header received
0808 04:34:59.316564 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:59.316585 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:59.317025 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:59.317381 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:59.318551 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:59.318560 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:59.318579 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:34:59.318597 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:59.318681 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:34:59.318738 (+ 57us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:59.318909 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:59.319146 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 04:34:59.319424 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":200,"threads_started":1}
I20260808 04:34:59.322542 22958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b: Bootstrap starting.
I20260808 04:34:59.324919 22958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b: Neither blocks nor log segments found. Creating new log.
I20260808 04:34:59.325659 22958 log.cc:824] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b: Log is configured to *not* fsync() on all Append() calls
I20260808 04:34:59.327900 22958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b: No bootstrap required, opened a new log
I20260808 04:34:59.331424 22958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0ff66ee23e4ab09faede8f0b9b965b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39741 } }
I20260808 04:34:59.331682 22958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:34:59.331766 22958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c0ff66ee23e4ab09faede8f0b9b965b, State: Initialized, Role: FOLLOWER
I20260808 04:34:59.332185 22958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0ff66ee23e4ab09faede8f0b9b965b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39741 } }
I20260808 04:34:59.332346 22958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:34:59.332433 22958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:34:59.332553 22958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:34:59.333491 22958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0ff66ee23e4ab09faede8f0b9b965b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39741 } }
I20260808 04:34:59.333832 22958 leader_election.cc:302] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c0ff66ee23e4ab09faede8f0b9b965b; no voters:
I20260808 04:34:59.334192 22958 leader_election.cc:288] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:34:59.334430 22965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:34:59.334729 22965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b [term 1 LEADER]: Becoming Leader. State: Replica: 0c0ff66ee23e4ab09faede8f0b9b965b, State: Running, Role: LEADER
I20260808 04:34:59.335070 22965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0ff66ee23e4ab09faede8f0b9b965b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39741 } }
I20260808 04:34:59.335523 22958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:34:59.336884 22968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c0ff66ee23e4ab09faede8f0b9b965b. Latest consensus state: current_term: 1 leader_uuid: "0c0ff66ee23e4ab09faede8f0b9b965b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0ff66ee23e4ab09faede8f0b9b965b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39741 } } }
I20260808 04:34:59.336889 22967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c0ff66ee23e4ab09faede8f0b9b965b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0ff66ee23e4ab09faede8f0b9b965b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39741 } } }
I20260808 04:34:59.337014 22968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b [sys.catalog]: This master's current role is: LEADER
I20260808 04:34:59.337023 22967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b [sys.catalog]: This master's current role is: LEADER
I20260808 04:34:59.337441 22977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:34:59.340983 22977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:34:59.349457 22977 catalog_manager.cc:1347] Generated new cluster ID: 53c90a96e44c4e47b4e2786138ae9c97
I20260808 04:34:59.349555 22977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:34:59.370652 22977 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:34:59.371057 22977 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:34:59.380060 22977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b: Generated new TSK 0
I20260808 04:34:59.381242 22977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:34:59.390793 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:39741
--builtin_ntp_servers=127.15.228.84:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 04:34:59.557310 22990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:59.557615 22990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:59.557693 22990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:59.563733 22990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:59.563881 22990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:59.564002 22990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:34:59.571390 22990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:39741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:59.573027 22990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:59.574987 22990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:59.587877 23003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:59.587877 22997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:59.588074 22990 server_base.cc:1034] running on GCE node
W20260808 04:34:59.588332 22999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:59.588820 22990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:59.589507 22990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:59.590838 22990 hybrid_clock.cc:648] HybridClock initialized: now 1786163699590807 us; error 65 us; skew 500 ppm
I20260808 04:34:59.593113 22990 webserver.cc:460] Webserver started at http://127.15.228.65:40277/ using document root <none> and password file <none>
I20260808 04:34:59.593758 22990 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:59.593858 22990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:59.594105 22990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:59.596470 22990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "e29114286efa412485882a32bde3fc61"
format_stamp: "Formatted at 2026-08-08 04:34:59 on dist-test-slave-drl0"
server_key: "d89c46358a9ded5ab2ae7539a6cdfc74"
server_key_iv: "934bce0a0395bcb5c0ea2eb4d048e096"
server_key_version: "encryptionkey@0"
I20260808 04:34:59.597071 22990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "e29114286efa412485882a32bde3fc61"
format_stamp: "Formatted at 2026-08-08 04:34:59 on dist-test-slave-drl0"
server_key: "d89c46358a9ded5ab2ae7539a6cdfc74"
server_key_iv: "934bce0a0395bcb5c0ea2eb4d048e096"
server_key_version: "encryptionkey@0"
I20260808 04:34:59.601466 22990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:34:59.605307 23015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:59.607072 22990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:34:59.607231 22990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "e29114286efa412485882a32bde3fc61"
format_stamp: "Formatted at 2026-08-08 04:34:59 on dist-test-slave-drl0"
server_key: "d89c46358a9ded5ab2ae7539a6cdfc74"
server_key_iv: "934bce0a0395bcb5c0ea2eb4d048e096"
server_key_version: "encryptionkey@0"
I20260808 04:34:59.607362 22990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:59.612260 22990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:59.613061 22990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:59.613272 22990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:59.613862 22990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:34:59.614871 22990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:59.614918 22990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:59.615020 22990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:59.615068 22990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:59.626648 22990 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:42717
I20260808 04:34:59.626760 23181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:42717 every 8 connection(s)
I20260808 04:34:59.627753 22990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:34:59.632500 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 22990
I20260808 04:34:59.632665 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:34:59.633261 16273 external_mini_cluster.cc:1428] Setting key f2b66c1fa0b7c77098845f138ce7d65e
I20260808 04:34:59.635809 22963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:59.630071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:34087 (local address 127.15.228.126:39741)
0808 04:34:59.630266 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:59.630270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:59.631462 (+ 1192us) server_negotiation.cc:407] Connection header received
0808 04:34:59.633249 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:59.633253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:59.633306 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:59.633421 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:59.635046 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:59.635053 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:59.635056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:34:59.635117 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:59.635159 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:34:59.635174 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:59.635331 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:59.635649 (+ 318us) server_negotiation.cc:299] Negotiation successful
0808 04:34:59.635700 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260808 04:34:59.636015 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:39741
--builtin_ntp_servers=127.15.228.84:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 04:34:59.636914 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:59.630469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:39741 (local address 127.15.228.65:34087)
0808 04:34:59.631297 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:59.631339 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:59.632671 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:59.633669 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:59.633677 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:59.633765 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:59.634535 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:34:59.634536 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:59.634579 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:34:59.634714 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:59.635336 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:59.635347 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:59.635353 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 04:34:59.635561 (+ 208us) client_negotiation.cc:240] Negotiation successful
0808 04:34:59.635823 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":132,"threads_started":1}
I20260808 04:34:59.639070 23182 heartbeater.cc:347] Connected to a master server at 127.15.228.126:39741
I20260808 04:34:59.639326 23182 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:59.639794 23182 heartbeater.cc:511] Master 127.15.228.126:39741 requested a full tablet report, sending...
I20260808 04:34:59.641494 22909 ts_manager.cc:194] Registered new tserver with Master: e29114286efa412485882a32bde3fc61 (127.15.228.65:42717)
W20260808 04:34:59.771111 23186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:34:59.771442 23186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:34:59.771520 23186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:34:59.777292 23186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:34:59.777421 23186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:34:59.777535 23186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:34:59.782976 23186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:39741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:34:59.784837 23186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:34:59.786366 23186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:34:59.797106 23196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:34:59.797231 23197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:59.797322 23186 server_base.cc:1034] running on GCE node
W20260808 04:34:59.797194 23200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:34:59.798332 23186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:34:59.799543 23186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:34:59.800880 23186 hybrid_clock.cc:648] HybridClock initialized: now 1786163699800822 us; error 61 us; skew 500 ppm
I20260808 04:34:59.803813 23186 webserver.cc:460] Webserver started at http://127.15.228.66:38919/ using document root <none> and password file <none>
I20260808 04:34:59.804667 23186 fs_manager.cc:359] Metadata directory not provided
I20260808 04:34:59.804775 23186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:34:59.805058 23186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:34:59.807871 23186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "713530d9f8464439972eb301a4694e2e"
format_stamp: "Formatted at 2026-08-08 04:34:59 on dist-test-slave-drl0"
server_key: "914a8f87dda3a2f2f23c205c5fb429d3"
server_key_iv: "357b2d11503f253c7e1cffaf80b10ed4"
server_key_version: "encryptionkey@0"
I20260808 04:34:59.808554 23186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "713530d9f8464439972eb301a4694e2e"
format_stamp: "Formatted at 2026-08-08 04:34:59 on dist-test-slave-drl0"
server_key: "914a8f87dda3a2f2f23c205c5fb429d3"
server_key_iv: "357b2d11503f253c7e1cffaf80b10ed4"
server_key_version: "encryptionkey@0"
I20260808 04:34:59.813632 23186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:34:59.816737 23210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:59.818643 23186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:34:59.818838 23186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "713530d9f8464439972eb301a4694e2e"
format_stamp: "Formatted at 2026-08-08 04:34:59 on dist-test-slave-drl0"
server_key: "914a8f87dda3a2f2f23c205c5fb429d3"
server_key_iv: "357b2d11503f253c7e1cffaf80b10ed4"
server_key_version: "encryptionkey@0"
I20260808 04:34:59.819099 23186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:34:59.825753 23186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:34:59.826431 23186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:34:59.826660 23186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:34:59.827346 23186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:34:59.828306 23186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:34:59.828356 23186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:59.828543 23186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:34:59.828586 23186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:34:59.839447 23186 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:34579
I20260808 04:34:59.839471 23371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:34579 every 8 connection(s)
I20260808 04:34:59.840402 23186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:34:59.846103 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 23186
I20260808 04:34:59.846246 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:34:59.846625 16273 external_mini_cluster.cc:1428] Setting key bb60a5adf78988d8d8160a76759e03f9
I20260808 04:34:59.847977 22963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:59.842478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:53617 (local address 127.15.228.126:39741)
0808 04:34:59.842760 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0808 04:34:59.842766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:34:59.843916 (+ 1150us) server_negotiation.cc:407] Connection header received
0808 04:34:59.845428 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:34:59.845431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:34:59.845476 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:34:59.845562 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:34:59.847130 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:34:59.847136 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:34:59.847141 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:34:59.847157 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:34:59.847181 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:34:59.847193 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:34:59.847317 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 04:34:59.847792 (+ 475us) server_negotiation.cc:299] Negotiation successful
0808 04:34:59.847843 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260808 04:34:59.849668 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:39741
--builtin_ntp_servers=127.15.228.84:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 04:34:59.849686 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:34:59.842875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:39741 (local address 127.15.228.66:53617)
0808 04:34:59.843723 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0808 04:34:59.843771 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 04:34:59.845168 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:34:59.845838 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:34:59.845849 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:34:59.845980 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:34:59.846697 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:34:59.846699 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:34:59.846772 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:34:59.846875 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:34:59.847439 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:34:59.847465 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:34:59.847477 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 04:34:59.847816 (+ 339us) client_negotiation.cc:240] Negotiation successful
0808 04:34:59.848215 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":141,"threads_started":1}
I20260808 04:34:59.851048 23373 heartbeater.cc:347] Connected to a master server at 127.15.228.126:39741
I20260808 04:34:59.851333 23373 heartbeater.cc:464] Registering TS with master...
I20260808 04:34:59.851920 23373 heartbeater.cc:511] Master 127.15.228.126:39741 requested a full tablet report, sending...
I20260808 04:34:59.853259 22909 ts_manager.cc:194] Registered new tserver with Master: 713530d9f8464439972eb301a4694e2e (127.15.228.66:34579)
W20260808 04:35:00.000427 23380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:00.000828 23380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:00.000881 23380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:00.007262 23380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:00.007364 23380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:00.007463 23380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:35:00.012756 23380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:39741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:00.013988 23380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:00.015529 23380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:00.026489 23388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:00.026536 23389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:00.026502 23392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:00.026568 23380 server_base.cc:1034] running on GCE node
I20260808 04:35:00.027390 23380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:00.028518 23380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:00.029801 23380 hybrid_clock.cc:648] HybridClock initialized: now 1786163700029776 us; error 51 us; skew 500 ppm
I20260808 04:35:00.031976 23380 webserver.cc:460] Webserver started at http://127.15.228.67:41409/ using document root <none> and password file <none>
I20260808 04:35:00.032641 23380 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:00.032742 23380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:00.032997 23380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:00.035249 23380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "b6973799812146cf9b47629901f13674"
format_stamp: "Formatted at 2026-08-08 04:35:00 on dist-test-slave-drl0"
server_key: "d418a59499721ac756db473e4416cc49"
server_key_iv: "8fba5c87b94b5c8c88d89e64d962c8ad"
server_key_version: "encryptionkey@0"
I20260808 04:35:00.035869 23380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "b6973799812146cf9b47629901f13674"
format_stamp: "Formatted at 2026-08-08 04:35:00 on dist-test-slave-drl0"
server_key: "d418a59499721ac756db473e4416cc49"
server_key_iv: "8fba5c87b94b5c8c88d89e64d962c8ad"
server_key_version: "encryptionkey@0"
I20260808 04:35:00.040911 23380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 04:35:00.044695 23402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:00.046388 23380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:35:00.046566 23380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "b6973799812146cf9b47629901f13674"
format_stamp: "Formatted at 2026-08-08 04:35:00 on dist-test-slave-drl0"
server_key: "d418a59499721ac756db473e4416cc49"
server_key_iv: "8fba5c87b94b5c8c88d89e64d962c8ad"
server_key_version: "encryptionkey@0"
I20260808 04:35:00.046891 23380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:00.052356 23380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:00.053022 23380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:00.053241 23380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:00.053828 23380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:00.054819 23380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:00.054908 23380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:00.055004 23380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:00.055068 23380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:00.065950 23380 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:42027
I20260808 04:35:00.065985 23564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:42027 every 8 connection(s)
I20260808 04:35:00.066978 23380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:35:00.070148 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 23380
I20260808 04:35:00.070256 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:35:00.070605 16273 external_mini_cluster.cc:1428] Setting key fe328fbeb35830ed7cf16d146e3ce663
I20260808 04:35:00.075656 22963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:00.069743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:51375 (local address 127.15.228.126:39741)
0808 04:35:00.070012 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:00.070019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:00.071645 (+ 1626us) server_negotiation.cc:407] Connection header received
0808 04:35:00.073631 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:00.073634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:00.073676 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:00.073750 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:00.075143 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:00.075146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:00.075148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:35:00.075159 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:00.075179 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:35:00.075186 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:00.075380 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:00.075506 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 04:35:00.075546 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260808 04:35:00.076701 23568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:00.070164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:39741 (local address 127.15.228.67:51375)
0808 04:35:00.071383 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:00.071453 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:00.073333 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:00.073996 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:00.074005 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:00.074118 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:00.074816 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:35:00.074818 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:00.074861 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:35:00.074983 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:00.075367 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:00.075376 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:00.075382 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 04:35:00.075540 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 04:35:00.075830 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":206,"threads_started":1}
I20260808 04:35:00.078673 23565 heartbeater.cc:347] Connected to a master server at 127.15.228.126:39741
I20260808 04:35:00.079018 23565 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:00.079754 23565 heartbeater.cc:511] Master 127.15.228.126:39741 requested a full tablet report, sending...
I20260808 04:35:00.081213 22909 ts_manager.cc:194] Registered new tserver with Master: b6973799812146cf9b47629901f13674 (127.15.228.67:42027)
I20260808 04:35:00.085565 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:35:00.090251 22963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:00.088472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46282 (local address 127.15.228.126:39741)
0808 04:35:00.088611 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:00.088615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:00.088766 (+ 151us) server_negotiation.cc:407] Connection header received
0808 04:35:00.088822 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:00.088824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:00.088859 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:00.088937 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:00.089826 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:00.089830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:00.089832 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:35:00.089844 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:00.089862 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:35:00.089871 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:00.089963 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:00.090101 (+ 138us) server_negotiation.cc:299] Negotiation successful
0808 04:35:00.090139 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:35:00.093681 22909 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:35:00.097170 22909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:35:00.113535 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:00.109752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:34579 (local address 127.0.0.1:34342)
0808 04:35:00.111011 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:00.111067 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:00.111187 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:00.111757 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:00.111763 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:00.111843 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:00.112639 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:35:00.112643 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:00.112666 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:35:00.112743 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:00.113117 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:00.113128 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:00.113134 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 04:35:00.113272 (+ 138us) client_negotiation.cc:240] Negotiation successful
0808 04:35:00.113355 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"thread_start_us":76,"threads_started":1}
I20260808 04:35:00.113567 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:00.109884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:42027 (local address 127.0.0.1:40046)
0808 04:35:00.111098 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:00.111116 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:00.111212 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:00.112152 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:00.112159 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:00.112181 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:00.112865 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:35:00.112868 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:00.112881 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:35:00.112915 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:00.113355 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:00.113359 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:00.113362 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 04:35:00.113435 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 04:35:00.113465 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1138,"thread_start_us":269,"threads_started":1}
I20260808 04:35:00.114080 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:00.110683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54602 (local address 127.15.228.65:42717)
0808 04:35:00.111122 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:00.111131 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:00.111349 (+ 218us) server_negotiation.cc:407] Connection header received
0808 04:35:00.111534 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:00.111540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:00.111716 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:00.111851 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:00.113402 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:00.113408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:00.113418 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:35:00.113432 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:00.113472 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:35:00.113501 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:00.113585 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:00.113791 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 04:35:00.113888 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":104,"threads_started":1}
I20260808 04:35:00.114161 23583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:00.110084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34342 (local address 127.15.228.66:34579)
0808 04:35:00.110615 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:00.110630 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:00.111303 (+ 673us) server_negotiation.cc:407] Connection header received
0808 04:35:00.111373 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:00.111380 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:00.111603 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:00.111761 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:00.112896 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:00.112905 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:00.112920 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:35:00.112931 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:00.112997 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:35:00.113029 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:00.113146 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:00.113897 (+ 751us) server_negotiation.cc:299] Negotiation successful
0808 04:35:00.113985 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":155,"threads_started":1}
I20260808 04:35:00.114377 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:00.110777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.15.228.67:42027)
0808 04:35:00.111346 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:00.111358 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:00.111389 (+ 31us) server_negotiation.cc:407] Connection header received
0808 04:35:00.111437 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:00.111442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:00.111682 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:00.111838 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:00.113072 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:00.113080 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:00.113098 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:35:00.113114 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:00.113161 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:35:00.113192 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:00.113311 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:00.114149 (+ 838us) server_negotiation.cc:299] Negotiation successful
0808 04:35:00.114217 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":185,"threads_started":1}
I20260808 04:35:00.114392 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:00.110213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:42717 (local address 127.0.0.1:54602)
0808 04:35:00.111240 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:00.111255 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:00.111405 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:00.112588 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:00.112593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:00.112607 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:00.113226 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:35:00.113228 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:00.113240 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:35:00.113265 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:00.113663 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:00.113666 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:00.113669 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 04:35:00.114241 (+ 572us) client_negotiation.cc:240] Negotiation successful
0808 04:35:00.114277 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"thread_start_us":143,"threads_started":1}
I20260808 04:35:00.117321 23472 tablet_service.cc:1467] Processing CreateTablet for tablet 0ae9b47b19fe46d5aae1a65092b82462 (DEFAULT_TABLE table=test-table [id=caea06e6e1ad4a25ab56a4e87acee733]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:00.117326 23090 tablet_service.cc:1467] Processing CreateTablet for tablet 0ae9b47b19fe46d5aae1a65092b82462 (DEFAULT_TABLE table=test-table [id=caea06e6e1ad4a25ab56a4e87acee733]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:00.117328 23279 tablet_service.cc:1467] Processing CreateTablet for tablet 0ae9b47b19fe46d5aae1a65092b82462 (DEFAULT_TABLE table=test-table [id=caea06e6e1ad4a25ab56a4e87acee733]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:00.118496 23472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ae9b47b19fe46d5aae1a65092b82462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:00.118517 23279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ae9b47b19fe46d5aae1a65092b82462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:00.118517 23090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ae9b47b19fe46d5aae1a65092b82462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:00.126232 23594 tablet_bootstrap.cc:492] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674: Bootstrap starting.
I20260808 04:35:00.127550 23595 tablet_bootstrap.cc:492] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e: Bootstrap starting.
I20260808 04:35:00.127586 23599 tablet_bootstrap.cc:492] T 0ae9b47b19fe46d5aae1a65092b82462 P e29114286efa412485882a32bde3fc61: Bootstrap starting.
I20260808 04:35:00.128870 23594 tablet_bootstrap.cc:654] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:00.129374 23599 tablet_bootstrap.cc:654] T 0ae9b47b19fe46d5aae1a65092b82462 P e29114286efa412485882a32bde3fc61: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:00.129621 23595 tablet_bootstrap.cc:654] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:00.129848 23594 log.cc:824] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:00.130141 23599 log.cc:824] T 0ae9b47b19fe46d5aae1a65092b82462 P e29114286efa412485882a32bde3fc61: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:00.130410 23595 log.cc:824] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:00.132197 23599 tablet_bootstrap.cc:492] T 0ae9b47b19fe46d5aae1a65092b82462 P e29114286efa412485882a32bde3fc61: No bootstrap required, opened a new log
I20260808 04:35:00.132197 23594 tablet_bootstrap.cc:492] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674: No bootstrap required, opened a new log
I20260808 04:35:00.132349 23595 tablet_bootstrap.cc:492] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e: No bootstrap required, opened a new log
I20260808 04:35:00.132407 23599 ts_tablet_manager.cc:1397] T 0ae9b47b19fe46d5aae1a65092b82462 P e29114286efa412485882a32bde3fc61: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260808 04:35:00.132407 23594 ts_tablet_manager.cc:1397] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:35:00.132548 23595 ts_tablet_manager.cc:1397] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260808 04:35:00.137292 23595 raft_consensus.cc:348] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713530d9f8464439972eb301a4694e2e" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34579 } } peers { permanent_uuid: "b6973799812146cf9b47629901f13674" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42027 } } peers { permanent_uuid: "e29114286efa412485882a32bde3fc61" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 42717 } }
I20260808 04:35:00.137245 23594 raft_consensus.cc:348] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713530d9f8464439972eb301a4694e2e" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34579 } } peers { permanent_uuid: "b6973799812146cf9b47629901f13674" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42027 } } peers { permanent_uuid: "e29114286efa412485882a32bde3fc61" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 42717 } }
I20260808 04:35:00.137245 23599 raft_consensus.cc:348] T 0ae9b47b19fe46d5aae1a65092b82462 P e29114286efa412485882a32bde3fc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713530d9f8464439972eb301a4694e2e" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34579 } } peers { permanent_uuid: "b6973799812146cf9b47629901f13674" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42027 } } peers { permanent_uuid: "e29114286efa412485882a32bde3fc61" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 42717 } }
I20260808 04:35:00.137538 23599 raft_consensus.cc:374] T 0ae9b47b19fe46d5aae1a65092b82462 P e29114286efa412485882a32bde3fc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:00.137538 23594 raft_consensus.cc:374] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:00.137539 23595 raft_consensus.cc:374] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:00.137601 23599 raft_consensus.cc:729] T 0ae9b47b19fe46d5aae1a65092b82462 P e29114286efa412485882a32bde3fc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e29114286efa412485882a32bde3fc61, State: Initialized, Role: FOLLOWER
I20260808 04:35:00.137601 23594 raft_consensus.cc:729] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6973799812146cf9b47629901f13674, State: Initialized, Role: FOLLOWER
I20260808 04:35:00.137611 23595 raft_consensus.cc:729] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 713530d9f8464439972eb301a4694e2e, State: Initialized, Role: FOLLOWER
I20260808 04:35:00.138077 23599 consensus_queue.cc:260] T 0ae9b47b19fe46d5aae1a65092b82462 P e29114286efa412485882a32bde3fc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713530d9f8464439972eb301a4694e2e" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34579 } } peers { permanent_uuid: "b6973799812146cf9b47629901f13674" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42027 } } peers { permanent_uuid: "e29114286efa412485882a32bde3fc61" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 42717 } }
I20260808 04:35:00.138078 23595 consensus_queue.cc:260] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713530d9f8464439972eb301a4694e2e" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34579 } } peers { permanent_uuid: "b6973799812146cf9b47629901f13674" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42027 } } peers { permanent_uuid: "e29114286efa412485882a32bde3fc61" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 42717 } }
I20260808 04:35:00.138144 23594 consensus_queue.cc:260] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713530d9f8464439972eb301a4694e2e" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34579 } } peers { permanent_uuid: "b6973799812146cf9b47629901f13674" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42027 } } peers { permanent_uuid: "e29114286efa412485882a32bde3fc61" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 42717 } }
I20260808 04:35:00.139170 23599 ts_tablet_manager.cc:1428] T 0ae9b47b19fe46d5aae1a65092b82462 P e29114286efa412485882a32bde3fc61: Time spent starting tablet: real 0.007s user 0.003s sys 0.005s
I20260808 04:35:00.139170 23594 ts_tablet_manager.cc:1428] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260808 04:35:00.139297 23595 ts_tablet_manager.cc:1428] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 04:35:00.139547 23182 heartbeater.cc:503] Master 127.15.228.126:39741 was elected leader, sending a full tablet report...
I20260808 04:35:00.139633 23373 heartbeater.cc:503] Master 127.15.228.126:39741 was elected leader, sending a full tablet report...
I20260808 04:35:00.140839 23565 heartbeater.cc:503] Master 127.15.228.126:39741 was elected leader, sending a full tablet report...
W20260808 04:35:00.318099 23566 tablet.cc:2369] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:35:00.341723 23375 tablet.cc:2369] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:35:00.379549 23183 tablet.cc:2369] T 0ae9b47b19fe46d5aae1a65092b82462 P e29114286efa412485882a32bde3fc61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:35:00.484961 23605 raft_consensus.cc:482] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:35:00.485268 23605 raft_consensus.cc:504] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713530d9f8464439972eb301a4694e2e" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34579 } } peers { permanent_uuid: "b6973799812146cf9b47629901f13674" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42027 } } peers { permanent_uuid: "e29114286efa412485882a32bde3fc61" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 42717 } }
I20260808 04:35:00.486644 23605 leader_election.cc:288] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 713530d9f8464439972eb301a4694e2e (127.15.228.66:34579), e29114286efa412485882a32bde3fc61 (127.15.228.65:42717)
I20260808 04:35:00.489636 23583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:00.487325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:45605 (local address 127.15.228.66:34579)
0808 04:35:00.487494 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:00.487498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:00.488115 (+ 617us) server_negotiation.cc:407] Connection header received
0808 04:35:00.488183 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:00.488189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:00.488241 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:00.488332 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:00.489161 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:00.489167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:00.489174 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:35:00.489185 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:00.489203 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:35:00.489211 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:00.489321 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:00.489443 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 04:35:00.489491 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 04:35:00.489650 23608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:00.487063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:34579 (local address 127.15.228.67:45605)
0808 04:35:00.487883 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:00.487897 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:00.487998 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:00.488420 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:00.488423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:00.488439 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:00.488981 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:35:00.488983 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:00.488994 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:35:00.489020 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:00.489327 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:00.489330 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:00.489334 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 04:35:00.489413 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 04:35:00.489478 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732}
I20260808 04:35:00.489703 23568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:00.487063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:42717 (local address 127.15.228.67:36403)
0808 04:35:00.488130 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:00.488149 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:00.488243 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:00.488548 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:00.488550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:00.488562 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:00.489160 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:35:00.489162 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:00.489170 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:35:00.489193 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:00.489481 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:00.489484 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:00.489486 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 04:35:00.489562 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 04:35:00.489589 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":242,"thread_start_us":243,"threads_started":1}
I20260808 04:35:00.489739 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:00.487228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:36403 (local address 127.15.228.65:42717)
0808 04:35:00.487465 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:00.487471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:00.488258 (+ 787us) server_negotiation.cc:407] Connection header received
0808 04:35:00.488381 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:00.488384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:00.488426 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:00.488518 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:00.489317 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:00.489320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:00.489323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:35:00.489337 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:00.489358 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:35:00.489369 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:00.489486 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:00.489569 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 04:35:00.489606 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:35:00.490476 23301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ae9b47b19fe46d5aae1a65092b82462" candidate_uuid: "b6973799812146cf9b47629901f13674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "713530d9f8464439972eb301a4694e2e" is_pre_election: true
I20260808 04:35:00.490477 23118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ae9b47b19fe46d5aae1a65092b82462" candidate_uuid: "b6973799812146cf9b47629901f13674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e29114286efa412485882a32bde3fc61" is_pre_election: true
I20260808 04:35:00.490815 23301 raft_consensus.cc:2454] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6973799812146cf9b47629901f13674 in term 0.
I20260808 04:35:00.490815 23118 raft_consensus.cc:2454] T 0ae9b47b19fe46d5aae1a65092b82462 P e29114286efa412485882a32bde3fc61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6973799812146cf9b47629901f13674 in term 0.
I20260808 04:35:00.491447 23404 leader_election.cc:302] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 713530d9f8464439972eb301a4694e2e, b6973799812146cf9b47629901f13674, e29114286efa412485882a32bde3fc61; no voters:
I20260808 04:35:00.491916 23605 raft_consensus.cc:2781] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:35:00.491999 23605 raft_consensus.cc:482] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:35:00.492031 23605 raft_consensus.cc:3037] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:00.493170 23605 raft_consensus.cc:504] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713530d9f8464439972eb301a4694e2e" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34579 } } peers { permanent_uuid: "b6973799812146cf9b47629901f13674" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42027 } } peers { permanent_uuid: "e29114286efa412485882a32bde3fc61" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 42717 } }
I20260808 04:35:00.493538 23605 leader_election.cc:288] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [CANDIDATE]: Term 1 election: Requested vote from peers 713530d9f8464439972eb301a4694e2e (127.15.228.66:34579), e29114286efa412485882a32bde3fc61 (127.15.228.65:42717)
I20260808 04:35:00.493966 23118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ae9b47b19fe46d5aae1a65092b82462" candidate_uuid: "b6973799812146cf9b47629901f13674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e29114286efa412485882a32bde3fc61"
I20260808 04:35:00.493958 23301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ae9b47b19fe46d5aae1a65092b82462" candidate_uuid: "b6973799812146cf9b47629901f13674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "713530d9f8464439972eb301a4694e2e"
I20260808 04:35:00.494102 23301 raft_consensus.cc:3037] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:00.494102 23118 raft_consensus.cc:3037] T 0ae9b47b19fe46d5aae1a65092b82462 P e29114286efa412485882a32bde3fc61 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:00.495508 23301 raft_consensus.cc:2454] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6973799812146cf9b47629901f13674 in term 1.
I20260808 04:35:00.495508 23118 raft_consensus.cc:2454] T 0ae9b47b19fe46d5aae1a65092b82462 P e29114286efa412485882a32bde3fc61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6973799812146cf9b47629901f13674 in term 1.
I20260808 04:35:00.495838 23404 leader_election.cc:302] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 713530d9f8464439972eb301a4694e2e, b6973799812146cf9b47629901f13674, e29114286efa412485882a32bde3fc61; no voters:
I20260808 04:35:00.496065 23605 raft_consensus.cc:2781] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:35:00.496322 23605 raft_consensus.cc:686] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [term 1 LEADER]: Becoming Leader. State: Replica: b6973799812146cf9b47629901f13674, State: Running, Role: LEADER
I20260808 04:35:00.496683 23605 consensus_queue.cc:237] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713530d9f8464439972eb301a4694e2e" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34579 } } peers { permanent_uuid: "b6973799812146cf9b47629901f13674" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42027 } } peers { permanent_uuid: "e29114286efa412485882a32bde3fc61" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 42717 } }
I20260808 04:35:00.501320 22908 catalog_manager.cc:5495] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6973799812146cf9b47629901f13674 (127.15.228.67). New cstate: current_term: 1 leader_uuid: "b6973799812146cf9b47629901f13674" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "713530d9f8464439972eb301a4694e2e" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6973799812146cf9b47629901f13674" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e29114286efa412485882a32bde3fc61" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 42717 } health_report { overall_health: UNKNOWN } } }
I20260808 04:35:00.573973 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:00.572223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.15.228.67:42027)
0808 04:35:00.572391 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:00.572395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:00.572410 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:35:00.572501 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:00.572503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:00.572570 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:00.572651 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:00.573494 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:00.573497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:00.573499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:35:00.573510 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:00.573529 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:35:00.573537 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:00.573634 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:00.573812 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 04:35:00.573862 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 04:35:00.580960 23301 raft_consensus.cc:1261] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e [term 1 FOLLOWER]: Refusing update from remote peer b6973799812146cf9b47629901f13674: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:35:00.580979 23118 raft_consensus.cc:1261] T 0ae9b47b19fe46d5aae1a65092b82462 P e29114286efa412485882a32bde3fc61 [term 1 FOLLOWER]: Refusing update from remote peer b6973799812146cf9b47629901f13674: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:35:00.581799 23610 consensus_queue.cc:1035] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e29114286efa412485882a32bde3fc61" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 42717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:35:00.582013 23605 consensus_queue.cc:1035] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [LEADER]: Connected to new peer: Peer: permanent_uuid: "713530d9f8464439972eb301a4694e2e" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:35:00.589378 23613 mvcc.cc:205] Tried to move back new op lower bound from 7316126517570510848 to 7316126517237968896. Current Snapshot: MvccSnapshot[applied={T|T < 7316126517570510848}]
I20260808 04:35:00.591151 23615 mvcc.cc:205] Tried to move back new op lower bound from 7316126517570510848 to 7316126517237968896. Current Snapshot: MvccSnapshot[applied={T|T < 7316126517570510848}]
I20260808 04:35:00.592600 23614 mvcc.cc:205] Tried to move back new op lower bound from 7316126517570510848 to 7316126517237968896. Current Snapshot: MvccSnapshot[applied={T|T < 7316126517570510848 or (T in {7316126517570510848})}]
I20260808 04:35:00.598713 22908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:35:00.599086 22908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:35:00.601553 22908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c0ff66ee23e4ab09faede8f0b9b965b: Sending DeleteTablet for 3 replicas of tablet 0ae9b47b19fe46d5aae1a65092b82462
I20260808 04:35:00.602640 23279 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ae9b47b19fe46d5aae1a65092b82462 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:00 UTC) from {username='slave'} at 127.0.0.1:34342
I20260808 04:35:00.602643 23472 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ae9b47b19fe46d5aae1a65092b82462 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:00 UTC) from {username='slave'} at 127.0.0.1:40046
I20260808 04:35:00.603188 23090 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ae9b47b19fe46d5aae1a65092b82462 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:00 UTC) from {username='slave'} at 127.0.0.1:54602
I20260808 04:35:00.603453 23625 tablet_replica.cc:330] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674: stopping tablet replica
I20260808 04:35:00.603478 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 22990
I20260808 04:35:00.603814 23625 raft_consensus.cc:2229] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:35:00.606669 23625 raft_consensus.cc:2258] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:00.603188 23624 tablet_replica.cc:330] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e: stopping tablet replica
I20260808 04:35:00.609676 23624 raft_consensus.cc:2229] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:35:00.610035 23624 raft_consensus.cc:2258] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:00.610667 23625 ts_tablet_manager.cc:1905] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:00.611737 23624 ts_tablet_manager.cc:1905] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:00.613626 23625 ts_tablet_manager.cc:1918] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:35:00.613729 23625 log.cc:1196] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/0ae9b47b19fe46d5aae1a65092b82462
I20260808 04:35:00.614061 23625 ts_tablet_manager.cc:1939] T 0ae9b47b19fe46d5aae1a65092b82462 P b6973799812146cf9b47629901f13674: Deleting consensus metadata
I20260808 04:35:00.615504 22885 catalog_manager.cc:4841] TS b6973799812146cf9b47629901f13674 (127.15.228.67:42027): tablet 0ae9b47b19fe46d5aae1a65092b82462 (table test-table [id=caea06e6e1ad4a25ab56a4e87acee733]) successfully deleted
I20260808 04:35:00.615883 23624 ts_tablet_manager.cc:1918] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:35:00.615993 23624 log.cc:1196] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/0ae9b47b19fe46d5aae1a65092b82462
I20260808 04:35:00.616334 23624 ts_tablet_manager.cc:1939] T 0ae9b47b19fe46d5aae1a65092b82462 P 713530d9f8464439972eb301a4694e2e: Deleting consensus metadata
I20260808 04:35:00.617717 22882 catalog_manager.cc:4841] TS 713530d9f8464439972eb301a4694e2e (127.15.228.66:34579): tablet 0ae9b47b19fe46d5aae1a65092b82462 (table test-table [id=caea06e6e1ad4a25ab56a4e87acee733]) successfully deleted
W20260808 04:35:00.618271 22883 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.228.65:42717 (error 108)
I20260808 04:35:00.619117 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 23186
I20260808 04:35:00.619691 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:00.619089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:42717 (local address 127.0.0.1:54608)
0808 04:35:00.619353 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:00.619426 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.65:42717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":70,"threads_started":1}
W20260808 04:35:00.623332 22883 catalog_manager.cc:4584] TS e29114286efa412485882a32bde3fc61 (127.15.228.65:42717): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ae9b47b19fe46d5aae1a65092b82462: Network error: Client connection negotiation failed: client connection to 127.15.228.65:42717: connect: Connection refused (error 111)
I20260808 04:35:00.639812 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 23380
I20260808 04:35:00.651870 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 22860
2026-08-08T04:35:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1565 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 04:35:00.681315 16273 mini_oidc.cc:152] Starting JWKS server
I20260808 04:35:00.683887 16273 webserver.cc:460] Webserver started at https://127.0.0.1:39421/ using document root <none> and password file <none>
I20260808 04:35:00.684113 16273 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 04:35:00.684432 16273 webserver.cc:460] Webserver started at http://127.0.0.1:37693/ using document root <none> and password file <none>
2026-08-08T04:35:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:35:00Z Disabled control of system clock
I20260808 04:35:00.696252 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:46117
--webserver_interface=127.15.228.126
--webserver_port=0
--builtin_ntp_servers=127.15.228.84:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:46117
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39421/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786163668922094-16273-0/testchainca.cert with env {}
W20260808 04:35:00.814617 23636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:00.814900 23636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:00.815035 23636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:00.819237 23636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:35:00.819305 23636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:00.819322 23636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:00.819339 23636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:35:00.819356 23636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 04:35:00.819379 23636 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 04:35:00.819393 23636 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39421/jwks.json
I20260808 04:35:00.825286 23636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:46117
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:46117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39421/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786163668922094-16273-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:00.826602 23636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:00.827870 23636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:00.836629 23645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:00.836630 23642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:00.836656 23643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:00.836994 23636 server_base.cc:1034] running on GCE node
I20260808 04:35:00.837791 23636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:00.839622 23636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:00.841112 23636 hybrid_clock.cc:648] HybridClock initialized: now 1786163700841081 us; error 114 us; skew 500 ppm
I20260808 04:35:00.843937 23636 webserver.cc:460] Webserver started at http://127.15.228.126:34473/ using document root <none> and password file <none>
I20260808 04:35:00.844811 23636 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:00.844897 23636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:00.845139 23636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:00.848132 23636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "94e56a516cb24c20b9adf8f11ba7f979"
format_stamp: "Formatted at 2026-08-08 04:35:00 on dist-test-slave-drl0"
server_key: "4f5fb2951cc006ead5e4404872e2ff65"
server_key_iv: "a75a5287665d00893532e5017c460e08"
server_key_version: "encryptionkey@0"
I20260808 04:35:00.848922 23636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "94e56a516cb24c20b9adf8f11ba7f979"
format_stamp: "Formatted at 2026-08-08 04:35:00 on dist-test-slave-drl0"
server_key: "4f5fb2951cc006ead5e4404872e2ff65"
server_key_iv: "a75a5287665d00893532e5017c460e08"
server_key_version: "encryptionkey@0"
I20260808 04:35:00.854526 23636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:35:00.857594 23656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:00.859557 23636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:35:00.859679 23636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "94e56a516cb24c20b9adf8f11ba7f979"
format_stamp: "Formatted at 2026-08-08 04:35:00 on dist-test-slave-drl0"
server_key: "4f5fb2951cc006ead5e4404872e2ff65"
server_key_iv: "a75a5287665d00893532e5017c460e08"
server_key_version: "encryptionkey@0"
I20260808 04:35:00.859802 23636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:00.878512 23636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:00.879621 23636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:00.879841 23636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:00.889166 23636 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:46117
I20260808 04:35:00.889284 23727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:46117 every 8 connection(s)
I20260808 04:35:00.890264 23636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:35:00.894907 23728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:00.895254 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 23636
I20260808 04:35:00.895385 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:35:00.895689 16273 external_mini_cluster.cc:1428] Setting key 657598bf36ea2cc0ffce6a6258c8d54f
I20260808 04:35:00.903151 23728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979: Bootstrap starting.
I20260808 04:35:00.907366 23728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:00.907662 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:00.897221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38362 (local address 127.15.228.126:46117)
0808 04:35:00.898395 (+ 1174us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:00.898417 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:00.898458 (+ 41us) server_negotiation.cc:407] Connection header received
0808 04:35:00.899230 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:00.899250 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:00.899693 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:00.899978 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:00.900594 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:00.901706 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:00.903254 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:00.903826 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:00.903966 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:00.905148 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:00.905154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:00.905168 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:35:00.905646 (+ 478us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:00.905795 (+ 149us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:35:00.905834 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:00.905930 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:00.906199 (+ 269us) server_negotiation.cc:299] Negotiation successful
0808 04:35:00.906749 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":306,"threads_started":1}
I20260808 04:35:00.908587 23728 log.cc:824] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:00.911624 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:35:00.911830 23728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979: No bootstrap required, opened a new log
I20260808 04:35:00.916136 23728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94e56a516cb24c20b9adf8f11ba7f979" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 46117 } }
I20260808 04:35:00.916460 23728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:00.916545 23728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94e56a516cb24c20b9adf8f11ba7f979, State: Initialized, Role: FOLLOWER
I20260808 04:35:00.917058 23728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94e56a516cb24c20b9adf8f11ba7f979" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 46117 } }
I20260808 04:35:00.917249 23728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:35:00.917325 23728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:35:00.917454 23728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:00.918484 23728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94e56a516cb24c20b9adf8f11ba7f979" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 46117 } }
I20260808 04:35:00.918784 23728 leader_election.cc:302] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94e56a516cb24c20b9adf8f11ba7f979; no voters:
I20260808 04:35:00.919171 23728 leader_election.cc:288] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:35:00.919286 23740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:35:00.919472 23740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979 [term 1 LEADER]: Becoming Leader. State: Replica: 94e56a516cb24c20b9adf8f11ba7f979, State: Running, Role: LEADER
I20260808 04:35:00.919775 23740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94e56a516cb24c20b9adf8f11ba7f979" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 46117 } }
I20260808 04:35:00.920547 23728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:35:00.921581 23743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94e56a516cb24c20b9adf8f11ba7f979. Latest consensus state: current_term: 1 leader_uuid: "94e56a516cb24c20b9adf8f11ba7f979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94e56a516cb24c20b9adf8f11ba7f979" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 46117 } } }
I20260808 04:35:00.921721 23743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:00.922156 23750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:35:00.922128 23742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94e56a516cb24c20b9adf8f11ba7f979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94e56a516cb24c20b9adf8f11ba7f979" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 46117 } } }
I20260808 04:35:00.922333 23742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:00.925693 23750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:35:00.934413 23750 catalog_manager.cc:1347] Generated new cluster ID: 025d6cde3dbf48689cf4dfd00a969e92
I20260808 04:35:00.934549 23750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:35:00.943849 23750 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:35:00.945478 23750 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:35:00.956168 23750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94e56a516cb24c20b9adf8f11ba7f979: Generated new TSK 0
I20260808 04:35:00.956972 23750 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 04:35:00.997159 23777 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 04:35:00.997344 23777 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 04:35:00.998421 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:00.971061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38364 (local address 127.15.228.126:46117)
0808 04:35:00.971221 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:00.971227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:00.971484 (+ 257us) server_negotiation.cc:407] Connection header received
0808 04:35:00.971781 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:00.971788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:00.971876 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:00.971986 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0808 04:35:00.972338 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:00.972926 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:00.974169 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:00.974401 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:00.974490 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:00.974905 (+ 415us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 04:35:00.997712 (+ 22807us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 04:35:00.998227 (+ 515us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38364: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260808 04:35:00.998570 23737 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38364: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260808 04:35:01.000099 23776 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:00.970977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:46117 (local address 127.0.0.1:38364)
0808 04:35:00.971349 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:00.971366 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:00.971514 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:00.972025 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:00.972029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:00.972038 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0808 04:35:00.972179 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:00.972187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:00.973037 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:00.973040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:00.974029 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:00.974039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:00.974523 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:00.974527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:00.974620 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:00.974674 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 04:35:00.998079 (+ 23405us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 04:35:00.998381 (+ 302us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.228.126:46117: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":137,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.15.228.126:46117: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 04:35:01.001158 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 23636
2026-08-08T04:35:01Z chronyd exiting
I20260808 04:35:01.045953 16273 test_util.cc:175] -----------------------------------------------
I20260808 04:35:01.046054 16273 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786163668922094-16273-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (366 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 04:35:01.047360 16273 mini_oidc.cc:152] Starting JWKS server
I20260808 04:35:01.048267 16273 webserver.cc:460] Webserver started at https://127.0.0.1:44029/ using document root <none> and password file <none>
I20260808 04:35:01.048347 16273 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 04:35:01.048676 16273 webserver.cc:460] Webserver started at http://127.0.0.1:38737/ using document root <none> and password file <none>
2026-08-08T04:35:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:35:01Z Disabled control of system clock
I20260808 04:35:01.059729 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:38971
--webserver_interface=127.15.228.126
--webserver_port=0
--builtin_ntp_servers=127.15.228.84:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:38971
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44029/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786163668922094-16273-0/testchainca.cert with env {}
W20260808 04:35:01.191599 23789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:01.191891 23789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:01.191932 23789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:01.195981 23789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:35:01.196040 23789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:01.196056 23789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:01.196074 23789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:35:01.196089 23789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 04:35:01.196159 23789 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 04:35:01.196183 23789 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44029/jwks.json
I20260808 04:35:01.201798 23789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:38971
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:38971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44029/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786163668922094-16273-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:01.203032 23789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:01.204126 23789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:01.214178 23804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:01.214239 23797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:01.214404 23789 server_base.cc:1034] running on GCE node
W20260808 04:35:01.214395 23801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:01.215171 23789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:01.216346 23789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:01.217605 23789 hybrid_clock.cc:648] HybridClock initialized: now 1786163701217575 us; error 44 us; skew 500 ppm
I20260808 04:35:01.222481 23789 webserver.cc:460] Webserver started at http://127.15.228.126:42693/ using document root <none> and password file <none>
I20260808 04:35:01.223186 23789 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:01.223291 23789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:01.223536 23789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:01.225782 23789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "a971be416b584ec99256d24a625b2e3b"
format_stamp: "Formatted at 2026-08-08 04:35:01 on dist-test-slave-drl0"
server_key: "32aaa8d6280d23334df7e24c8f5ecdc5"
server_key_iv: "dc5c34e6e960035b9b939c2edbb27da7"
server_key_version: "encryptionkey@0"
I20260808 04:35:01.226405 23789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "a971be416b584ec99256d24a625b2e3b"
format_stamp: "Formatted at 2026-08-08 04:35:01 on dist-test-slave-drl0"
server_key: "32aaa8d6280d23334df7e24c8f5ecdc5"
server_key_iv: "dc5c34e6e960035b9b939c2edbb27da7"
server_key_version: "encryptionkey@0"
I20260808 04:35:01.230504 23789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:35:01.233438 23811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:01.234879 23789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:35:01.235064 23789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "a971be416b584ec99256d24a625b2e3b"
format_stamp: "Formatted at 2026-08-08 04:35:01 on dist-test-slave-drl0"
server_key: "32aaa8d6280d23334df7e24c8f5ecdc5"
server_key_iv: "dc5c34e6e960035b9b939c2edbb27da7"
server_key_version: "encryptionkey@0"
I20260808 04:35:01.235191 23789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:01.256381 23789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:01.257503 23789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:01.257777 23789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:01.266469 23789 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:38971
I20260808 04:35:01.266502 23880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:38971 every 8 connection(s)
I20260808 04:35:01.267513 23789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:35:01.269392 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 23789
I20260808 04:35:01.269527 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:35:01.269829 16273 external_mini_cluster.cc:1428] Setting key 188082fc0227091967ddc866a574e7ef
I20260808 04:35:01.271970 23881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:01.277702 23881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b: Bootstrap starting.
I20260808 04:35:01.279706 23891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:01.271670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45350 (local address 127.15.228.126:38971)
0808 04:35:01.272431 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:01.272448 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:01.272485 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:35:01.273147 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:01.273162 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:01.273558 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:01.273887 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:01.274327 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:01.275412 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:01.276318 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:01.276781 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:01.276965 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:01.277884 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:01.277891 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:01.277907 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:35:01.278317 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:01.278372 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:35:01.278393 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:01.278502 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:01.278673 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 04:35:01.278987 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":156,"threads_started":1}
I20260808 04:35:01.280320 23881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:01.281190 23881 log.cc:824] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:01.283093 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:35:01.283545 23881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b: No bootstrap required, opened a new log
I20260808 04:35:01.288340 23881 raft_consensus.cc:348] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a971be416b584ec99256d24a625b2e3b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 38971 } }
I20260808 04:35:01.288601 23881 raft_consensus.cc:374] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:01.288638 23881 raft_consensus.cc:729] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a971be416b584ec99256d24a625b2e3b, State: Initialized, Role: FOLLOWER
I20260808 04:35:01.289116 23881 consensus_queue.cc:260] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a971be416b584ec99256d24a625b2e3b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 38971 } }
I20260808 04:35:01.289248 23881 raft_consensus.cc:388] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:35:01.289332 23881 raft_consensus.cc:482] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:35:01.289431 23881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:01.290675 23881 raft_consensus.cc:504] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a971be416b584ec99256d24a625b2e3b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 38971 } }
I20260808 04:35:01.291074 23881 leader_election.cc:302] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a971be416b584ec99256d24a625b2e3b; no voters:
I20260808 04:35:01.291420 23881 leader_election.cc:288] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:35:01.291668 23896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:35:01.291898 23896 raft_consensus.cc:686] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b [term 1 LEADER]: Becoming Leader. State: Replica: a971be416b584ec99256d24a625b2e3b, State: Running, Role: LEADER
I20260808 04:35:01.292219 23896 consensus_queue.cc:237] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a971be416b584ec99256d24a625b2e3b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 38971 } }
I20260808 04:35:01.293116 23881 sys_catalog.cc:564] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:35:01.294057 23900 sys_catalog.cc:455] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a971be416b584ec99256d24a625b2e3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a971be416b584ec99256d24a625b2e3b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 38971 } } }
I20260808 04:35:01.294215 23900 sys_catalog.cc:458] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:01.294517 23901 sys_catalog.cc:455] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a971be416b584ec99256d24a625b2e3b. Latest consensus state: current_term: 1 leader_uuid: "a971be416b584ec99256d24a625b2e3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a971be416b584ec99256d24a625b2e3b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 38971 } } }
I20260808 04:35:01.294623 23901 sys_catalog.cc:458] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:01.294859 23910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:35:01.297847 23910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:35:01.304113 23910 catalog_manager.cc:1347] Generated new cluster ID: 24a48ae1b9aa446c8cb94ce5e145285b
I20260808 04:35:01.304194 23910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:35:01.319568 23910 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:35:01.321457 23910 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:35:01.327883 23910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a971be416b584ec99256d24a625b2e3b: Generated new TSK 0
I20260808 04:35:01.329080 23910 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 04:35:01.363324 23933 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 04:35:01.363451 23933 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 04:35:01.364125 23891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:01.346788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45358 (local address 127.15.228.126:38971)
0808 04:35:01.347012 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:01.347019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:01.347146 (+ 127us) server_negotiation.cc:407] Connection header received
0808 04:35:01.347348 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:01.347352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:01.347418 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:01.347565 (+ 147us) server_negotiation.cc:226] Negotiated authn=JWT
0808 04:35:01.347880 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:01.348719 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:01.349719 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:01.350073 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:01.350183 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:01.350538 (+ 355us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 04:35:01.363707 (+ 13169us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44029/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 04:35:01.363977 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45358: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44029/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":60}
W20260808 04:35:01.364195 23891 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45358: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44029/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260808 04:35:01.364277 23932 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:01.346655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:38971 (local address 127.0.0.1:45358)
0808 04:35:01.347010 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:01.347029 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:01.347191 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:01.347605 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:01.347608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:01.347617 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0808 04:35:01.347740 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:01.347745 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:01.348944 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:01.348948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:01.349598 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:01.349605 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:01.350241 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:01.350245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:01.350314 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:01.350343 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 04:35:01.363929 (+ 13586us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44029/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 04:35:01.364079 (+ 150us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.228.126:38971: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44029/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":360,"threads_started":1}
I20260808 04:35:01.365051 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 23789
2026-08-08T04:35:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (365 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 04:35:01.413115 16273 mini_oidc.cc:152] Starting JWKS server
I20260808 04:35:01.414436 16273 webserver.cc:460] Webserver started at https://127.0.0.1:36611/ using document root <none> and password file <none>
I20260808 04:35:01.414636 16273 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 04:35:01.415194 16273 webserver.cc:460] Webserver started at http://127.0.0.1:41561/ using document root <none> and password file <none>
2026-08-08T04:35:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:35:01Z Disabled control of system clock
I20260808 04:35:01.427922 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:43965
--webserver_interface=127.15.228.126
--webserver_port=0
--builtin_ntp_servers=127.15.228.84:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:43965
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36611/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 04:35:01.552393 23940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:01.552790 23940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:01.552839 23940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:01.558815 23940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:35:01.558976 23940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:01.558995 23940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:01.559016 23940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:35:01.559034 23940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 04:35:01.559053 23940 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 04:35:01.559067 23940 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36611/jwks.json
I20260808 04:35:01.565640 23940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:43965
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:43965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36611/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:01.566990 23940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:01.568954 23940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:01.578809 23947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:01.578958 23946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:01.578809 23951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:01.579106 23940 server_base.cc:1034] running on GCE node
I20260808 04:35:01.579787 23940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:01.580940 23940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:01.582165 23940 hybrid_clock.cc:648] HybridClock initialized: now 1786163701582155 us; error 42 us; skew 500 ppm
I20260808 04:35:01.584265 23940 webserver.cc:460] Webserver started at http://127.15.228.126:44521/ using document root <none> and password file <none>
I20260808 04:35:01.584926 23940 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:01.584996 23940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:01.585242 23940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:01.587569 23940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "16ceb941fc7942b685d28928c3b386d6"
format_stamp: "Formatted at 2026-08-08 04:35:01 on dist-test-slave-drl0"
server_key: "40415a36110abbe6f5350127a7382e5f"
server_key_iv: "12a71afc49941e216d2834f6b769cd59"
server_key_version: "encryptionkey@0"
I20260808 04:35:01.588193 23940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "16ceb941fc7942b685d28928c3b386d6"
format_stamp: "Formatted at 2026-08-08 04:35:01 on dist-test-slave-drl0"
server_key: "40415a36110abbe6f5350127a7382e5f"
server_key_iv: "12a71afc49941e216d2834f6b769cd59"
server_key_version: "encryptionkey@0"
I20260808 04:35:01.592216 23940 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:35:01.594763 23959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:01.596664 23940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:35:01.596937 23940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "16ceb941fc7942b685d28928c3b386d6"
format_stamp: "Formatted at 2026-08-08 04:35:01 on dist-test-slave-drl0"
server_key: "40415a36110abbe6f5350127a7382e5f"
server_key_iv: "12a71afc49941e216d2834f6b769cd59"
server_key_version: "encryptionkey@0"
I20260808 04:35:01.597136 23940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:01.612130 23940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:01.613215 23940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:01.613510 23940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:01.622370 23940 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:43965
I20260808 04:35:01.622572 24029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:43965 every 8 connection(s)
I20260808 04:35:01.623639 23940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:35:01.627090 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:01.627238 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 23940
I20260808 04:35:01.627349 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:35:01.627674 16273 external_mini_cluster.cc:1428] Setting key 6a6b701c3b2091ccdf1f2b0d8d120475
I20260808 04:35:01.633306 24030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6: Bootstrap starting.
I20260808 04:35:01.635689 24030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:01.636538 24030 log.cc:824] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:01.639007 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:01.629098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.15.228.126:43965)
0808 04:35:01.630152 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:01.630174 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:01.630234 (+ 60us) server_negotiation.cc:407] Connection header received
0808 04:35:01.631468 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:01.631492 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:01.631936 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:01.632289 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:01.632752 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:01.633826 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:01.634788 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:01.635344 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:01.635530 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:01.636833 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:01.636840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:01.636857 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:35:01.637314 (+ 457us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:01.637369 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:35:01.637390 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:01.637518 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:01.637682 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 04:35:01.637947 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":152,"threads_started":1}
I20260808 04:35:01.639139 24030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6: No bootstrap required, opened a new log
I20260808 04:35:01.642019 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:35:01.643462 24030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ceb941fc7942b685d28928c3b386d6" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43965 } }
I20260808 04:35:01.643683 24030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:01.643718 24030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ceb941fc7942b685d28928c3b386d6, State: Initialized, Role: FOLLOWER
I20260808 04:35:01.644207 24030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ceb941fc7942b685d28928c3b386d6" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43965 } }
I20260808 04:35:01.644336 24030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:35:01.644423 24030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:35:01.644541 24030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:01.645524 24030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ceb941fc7942b685d28928c3b386d6" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43965 } }
I20260808 04:35:01.645879 24030 leader_election.cc:302] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16ceb941fc7942b685d28928c3b386d6; no voters:
I20260808 04:35:01.646202 24030 leader_election.cc:288] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:35:01.646391 24039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:35:01.646631 24039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6 [term 1 LEADER]: Becoming Leader. State: Replica: 16ceb941fc7942b685d28928c3b386d6, State: Running, Role: LEADER
I20260808 04:35:01.647049 24039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ceb941fc7942b685d28928c3b386d6" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43965 } }
I20260808 04:35:01.647564 24030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:35:01.649034 24041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16ceb941fc7942b685d28928c3b386d6. Latest consensus state: current_term: 1 leader_uuid: "16ceb941fc7942b685d28928c3b386d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ceb941fc7942b685d28928c3b386d6" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43965 } } }
I20260808 04:35:01.649170 24041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:01.649384 24040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16ceb941fc7942b685d28928c3b386d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ceb941fc7942b685d28928c3b386d6" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43965 } } }
I20260808 04:35:01.649533 24040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:01.649647 24057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:35:01.652477 24057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:35:01.657981 24057 catalog_manager.cc:1347] Generated new cluster ID: 220c3e2c27184a48b1edfa5e5febee8c
I20260808 04:35:01.658085 24057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:35:01.694831 24057 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:35:01.696795 24057 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:35:01.703272 24057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16ceb941fc7942b685d28928c3b386d6: Generated new TSK 0
I20260808 04:35:01.703999 24057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:35:01.724927 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:01.716329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41962 (local address 127.15.228.126:43965)
0808 04:35:01.716488 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:01.716493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:01.716741 (+ 248us) server_negotiation.cc:407] Connection header received
0808 04:35:01.716878 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:01.716881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:01.716936 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:01.717033 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0808 04:35:01.717429 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:01.718133 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:01.719513 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:01.719893 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:01.720032 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:01.720425 (+ 393us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 04:35:01.724469 (+ 4044us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36611/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 04:35:01.724745 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41962: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36611/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":61}
W20260808 04:35:01.725005 24036 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41962: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36611/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260808 04:35:01.726902 24078 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:01.716270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:43965 (local address 127.0.0.1:41962)
0808 04:35:01.716608 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:01.716623 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:01.716762 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:01.717056 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:01.717059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:01.717068 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0808 04:35:01.717185 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:01.717191 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:01.718322 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:01.718326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:01.719371 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:01.719381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:01.720066 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:01.720073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:01.720182 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:01.720211 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 04:35:01.724672 (+ 4461us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36611/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 04:35:01.726699 (+ 2027us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.228.126:43965: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36611/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":1803,"thread_start_us":146,"threads_started":1}
I20260808 04:35:01.727535 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 23940
E20260808 04:35:01.748653 24080 webserver.cc:533] Webserver: sslize failed:
E20260808 04:35:01.748744 24080 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-08T04:35:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (370 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T04:35:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:35:01Z Disabled control of system clock
I20260808 04:35:01.794871 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:36441
--webserver_interface=127.15.228.126
--webserver_port=0
--builtin_ntp_servers=127.15.228.84:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:36441
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786163668922094-16273-0/testchainca.cert with env {}
W20260808 04:35:01.926443 24086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:01.926705 24086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:01.926748 24086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:01.930868 24086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:35:01.930950 24086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:01.930967 24086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:01.930984 24086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:35:01.930998 24086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 04:35:01.931015 24086 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 04:35:01.931028 24086 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 04:35:01.938679 24086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:36441
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:36441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786163668922094-16273-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:01.939875 24086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:01.940979 24086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:01.950536 24093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:01.950891 24096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:01.950918 24094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:01.951216 24086 server_base.cc:1034] running on GCE node
I20260808 04:35:01.951988 24086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:01.953502 24086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:01.954836 24086 hybrid_clock.cc:648] HybridClock initialized: now 1786163701954790 us; error 78 us; skew 500 ppm
I20260808 04:35:01.957113 24086 webserver.cc:460] Webserver started at http://127.15.228.126:42953/ using document root <none> and password file <none>
I20260808 04:35:01.957753 24086 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:01.957863 24086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:01.958106 24086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:01.960350 24086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "9c4225aa0f67462dadf38e4c3446aa2c"
format_stamp: "Formatted at 2026-08-08 04:35:01 on dist-test-slave-drl0"
server_key: "3487771a02f94b74d9dfcb6c81c449aa"
server_key_iv: "cad72b9d4456db06b104e03426dacb9d"
server_key_version: "encryptionkey@0"
I20260808 04:35:01.961023 24086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "9c4225aa0f67462dadf38e4c3446aa2c"
format_stamp: "Formatted at 2026-08-08 04:35:01 on dist-test-slave-drl0"
server_key: "3487771a02f94b74d9dfcb6c81c449aa"
server_key_iv: "cad72b9d4456db06b104e03426dacb9d"
server_key_version: "encryptionkey@0"
I20260808 04:35:01.965034 24086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:35:01.968116 24104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:01.969848 24086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:35:01.969995 24086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "9c4225aa0f67462dadf38e4c3446aa2c"
format_stamp: "Formatted at 2026-08-08 04:35:01 on dist-test-slave-drl0"
server_key: "3487771a02f94b74d9dfcb6c81c449aa"
server_key_iv: "cad72b9d4456db06b104e03426dacb9d"
server_key_version: "encryptionkey@0"
I20260808 04:35:01.970121 24086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:01.995311 24086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:01.996201 24086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:01.996403 24086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:02.005492 24086 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:36441
I20260808 04:35:02.005513 24179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:36441 every 8 connection(s)
I20260808 04:35:02.006970 24086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:35:02.011163 24180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:02.013618 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 24086
I20260808 04:35:02.013760 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:35:02.014073 16273 external_mini_cluster.cc:1428] Setting key 1ead5d3028d3615ef3f5e146abee6380
I20260808 04:35:02.018973 24180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c: Bootstrap starting.
I20260808 04:35:02.021657 24180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:02.022539 24180 log.cc:824] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:02.025760 24180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c: No bootstrap required, opened a new log
I20260808 04:35:02.026782 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:02.016082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54296 (local address 127.15.228.126:36441)
0808 04:35:02.016858 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:02.016882 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:02.016928 (+ 46us) server_negotiation.cc:407] Connection header received
0808 04:35:02.017767 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:02.017790 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:02.018271 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:02.018640 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:02.019370 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:02.020839 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:02.021959 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:02.022657 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:02.022815 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:02.024250 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:02.024260 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:02.024284 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:35:02.024927 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:02.025021 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:35:02.025145 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:02.025286 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:02.025514 (+ 228us) server_negotiation.cc:299] Negotiation successful
0808 04:35:02.025801 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":141,"threads_started":1}
I20260808 04:35:02.029908 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:35:02.030185 24180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4225aa0f67462dadf38e4c3446aa2c" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36441 } }
I20260808 04:35:02.030431 24180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:02.030510 24180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c4225aa0f67462dadf38e4c3446aa2c, State: Initialized, Role: FOLLOWER
I20260808 04:35:02.031040 24180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4225aa0f67462dadf38e4c3446aa2c" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36441 } }
I20260808 04:35:02.031327 24180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:35:02.031431 24180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:35:02.031538 24180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:02.032955 24180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4225aa0f67462dadf38e4c3446aa2c" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36441 } }
I20260808 04:35:02.033643 24180 leader_election.cc:302] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c4225aa0f67462dadf38e4c3446aa2c; no voters:
I20260808 04:35:02.034823 24180 leader_election.cc:288] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:35:02.035017 24187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:35:02.035290 24187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c [term 1 LEADER]: Becoming Leader. State: Replica: 9c4225aa0f67462dadf38e4c3446aa2c, State: Running, Role: LEADER
I20260808 04:35:02.035624 24187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4225aa0f67462dadf38e4c3446aa2c" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36441 } }
I20260808 04:35:02.036195 24180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:35:02.038024 24189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c4225aa0f67462dadf38e4c3446aa2c. Latest consensus state: current_term: 1 leader_uuid: "9c4225aa0f67462dadf38e4c3446aa2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4225aa0f67462dadf38e4c3446aa2c" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36441 } } }
I20260808 04:35:02.038192 24189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:02.039085 24200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:35:02.039085 24188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c4225aa0f67462dadf38e4c3446aa2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4225aa0f67462dadf38e4c3446aa2c" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36441 } } }
I20260808 04:35:02.039250 24188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:02.042343 24200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:35:02.049923 24200 catalog_manager.cc:1347] Generated new cluster ID: 5efd51390cc246fc82d2c25fad4f52fa
I20260808 04:35:02.050011 24200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:35:02.102867 24200 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:35:02.104120 24200 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:35:02.112938 24200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c4225aa0f67462dadf38e4c3446aa2c: Generated new TSK 0
I20260808 04:35:02.113667 24200 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:35:02.563161 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:02.133015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54312 (local address 127.15.228.126:36441)
0808 04:35:02.133259 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:02.133266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:02.133564 (+ 298us) server_negotiation.cc:407] Connection header received
0808 04:35:02.133715 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:02.133719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:02.133782 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:02.133922 (+ 140us) server_negotiation.cc:226] Negotiated authn=JWT
0808 04:35:02.134175 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:02.135042 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:02.136255 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:02.136492 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:02.136618 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:02.136973 (+ 355us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 04:35:02.562574 (+425601us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 04:35:02.562986 (+ 412us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54312: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":122}
W20260808 04:35:02.563262 24184 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54312: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260808 04:35:02.563314 24219 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:02.132975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36441 (local address 127.0.0.1:54312)
0808 04:35:02.133446 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:02.133460 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:02.133579 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:02.133941 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:02.133943 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:02.133950 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0808 04:35:02.134063 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:02.134068 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:02.135178 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:02.135181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:02.135832 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:02.135840 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:02.136636 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:02.136641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:02.136739 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:02.136763 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 04:35:02.562871 (+426108us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 04:35:02.563106 (+ 235us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.228.126:36441: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":145,"threads_started":1}
I20260808 04:35:02.564395 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 24086
2026-08-08T04:35:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (808 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (155 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (142 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:35:02 dist-test-slave-drl0 krb5kdc[24238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:35:02 dist-test-slave-drl0 krb5kdc[24238](info): set up 2 sockets
Aug 08 04:35:02 dist-test-slave-drl0 krb5kdc[24238](info): commencing operation
krb5kdc: starting...
W20260808 04:35:04.125586 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:35:04 dist-test-slave-drl0 krb5kdc[24238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:35:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:35:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:35:04.275524 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:42011
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:43719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:42011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:04.402745 24254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:04.403069 24254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:04.403111 24254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:04.407320 24254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:35:04.407382 24254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:04.407402 24254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:04.407418 24254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:35:04.407433 24254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:35:04.413318 24254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:43719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:42011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:42011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:04.414465 24254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:04.415990 24254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:04.425283 24264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:04.425253 24260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:04.425356 24254 server_base.cc:1034] running on GCE node
W20260808 04:35:04.425261 24261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:04.426146 24254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:04.427332 24254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:04.428547 24254 hybrid_clock.cc:648] HybridClock initialized: now 1786163704428518 us; error 40 us; skew 500 ppm
Aug 08 04:35:04 dist-test-slave-drl0 krb5kdc[24238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163704, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:04.432132 24254 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:35:04.433046 24254 webserver.cc:460] Webserver started at http://127.15.228.126:37335/ using document root <none> and password file <none>
I20260808 04:35:04.433643 24254 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:04.433740 24254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:04.433983 24254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:04.436208 24254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "a1907b8e186c4cccbd335c3257c8a304"
format_stamp: "Formatted at 2026-08-08 04:35:04 on dist-test-slave-drl0"
server_key: "e4aaa7d2e3ba989b812bd03a15d6b650"
server_key_iv: "3e4ab62b18a3f51959faa52565e21b95"
server_key_version: "encryptionkey@0"
I20260808 04:35:04.436796 24254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "a1907b8e186c4cccbd335c3257c8a304"
format_stamp: "Formatted at 2026-08-08 04:35:04 on dist-test-slave-drl0"
server_key: "e4aaa7d2e3ba989b812bd03a15d6b650"
server_key_iv: "3e4ab62b18a3f51959faa52565e21b95"
server_key_version: "encryptionkey@0"
I20260808 04:35:04.440826 24254 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:35:04.444483 24275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:04.446456 24254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:35:04.446607 24254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "a1907b8e186c4cccbd335c3257c8a304"
format_stamp: "Formatted at 2026-08-08 04:35:04 on dist-test-slave-drl0"
server_key: "e4aaa7d2e3ba989b812bd03a15d6b650"
server_key_iv: "3e4ab62b18a3f51959faa52565e21b95"
server_key_version: "encryptionkey@0"
I20260808 04:35:04.446766 24254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:04.462031 24254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:04.466621 24254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:04.466818 24254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:04.476346 24348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:42011 every 8 connection(s)
I20260808 04:35:04.476348 24254 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:42011
I20260808 04:35:04.477911 24254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:35:04.481194 24352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:04.485507 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 24254
I20260808 04:35:04.485658 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:35:04.485946 16273 external_mini_cluster.cc:1428] Setting key ce808df8c990b2b1ab01fa103ffc9c7a
I20260808 04:35:04.487628 24352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304: Bootstrap starting.
I20260808 04:35:04.491462 24352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:04.492483 24352 log.cc:824] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:04.494546 24352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304: No bootstrap required, opened a new log
Aug 08 04:35:04 dist-test-slave-drl0 krb5kdc[24238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:35:04.498677 24352 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1907b8e186c4cccbd335c3257c8a304" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42011 } }
I20260808 04:35:04.498980 24352 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:04.499073 24352 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1907b8e186c4cccbd335c3257c8a304, State: Initialized, Role: FOLLOWER
I20260808 04:35:04.499570 24352 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1907b8e186c4cccbd335c3257c8a304" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42011 } }
I20260808 04:35:04.499728 24352 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:35:04.499810 24352 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:35:04.499936 24352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:04.500881 24352 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1907b8e186c4cccbd335c3257c8a304" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42011 } }
I20260808 04:35:04.501253 24352 leader_election.cc:302] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1907b8e186c4cccbd335c3257c8a304; no voters:
I20260808 04:35:04.501678 24352 leader_election.cc:288] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:35:04.501786 24358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:35:04.502040 24358 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304 [term 1 LEADER]: Becoming Leader. State: Replica: a1907b8e186c4cccbd335c3257c8a304, State: Running, Role: LEADER
I20260808 04:35:04.502447 24358 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1907b8e186c4cccbd335c3257c8a304" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42011 } }
I20260808 04:35:04.502851 24355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:04.487573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56296 (local address 127.15.228.126:42011)
0808 04:35:04.488102 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:04.488118 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:04.488156 (+ 38us) server_negotiation.cc:407] Connection header received
0808 04:35:04.488812 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:04.488830 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:04.489199 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:04.489603 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:04.490479 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:04.491730 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:04.492935 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:04.493622 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:04.494044 (+ 422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:04.497136 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:04.497142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:04.497156 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:04.497185 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:04.499270 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:04.499779 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:04.499786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:04.499793 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:04.499951 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:04.500277 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:04.500282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:04.500284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:04.500736 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:04.501015 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:04.501312 (+ 297us) server_negotiation.cc:299] Negotiation successful
0808 04:35:04.501704 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":164,"threads_started":1}
I20260808 04:35:04.502898 24352 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:35:04.503880 24363 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1907b8e186c4cccbd335c3257c8a304. Latest consensus state: current_term: 1 leader_uuid: "a1907b8e186c4cccbd335c3257c8a304" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1907b8e186c4cccbd335c3257c8a304" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42011 } } }
I20260808 04:35:04.504010 24363 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:04.504817 24360 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1907b8e186c4cccbd335c3257c8a304" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1907b8e186c4cccbd335c3257c8a304" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42011 } } }
I20260808 04:35:04.504959 24360 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304 [sys.catalog]: This master's current role is: LEADER
W20260808 04:35:04.507824 24380 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:35:04.507893 24380 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:35:04.507992 24373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:35:04.508992 24373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:35:04.515002 24373 catalog_manager.cc:1347] Generated new cluster ID: 4c316eb286bf468995b3603933101cfe
I20260808 04:35:04.515072 24373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:35:04.531018 24373 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:35:04.532533 24373 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:35:04.546090 24373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1907b8e186c4cccbd335c3257c8a304: Generated new TSK 0
I20260808 04:35:04.546844 24373 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
I20260808 04:35:04.582919 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:42011
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:43719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:04.719715 24389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:04.720018 24389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:04.720095 24389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:04.724237 24389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:04.724315 24389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:04.724403 24389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:35:04.730355 24389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:43719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:42011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:04.731748 24389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:04.732987 24389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:04.743564 24400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:04.743705 24399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:04.743741 24403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:04.743924 24389 server_base.cc:1034] running on GCE node
I20260808 04:35:04.744370 24389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:04.745486 24389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:04.746750 24389 hybrid_clock.cc:648] HybridClock initialized: now 1786163704746720 us; error 43 us; skew 500 ppm
Aug 08 04:35:04 dist-test-slave-drl0 krb5kdc[24238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163704, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:04.749979 24389 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:35:04.751084 24389 webserver.cc:460] Webserver started at http://127.15.228.65:39151/ using document root <none> and password file <none>
I20260808 04:35:04.751700 24389 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:04.751768 24389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:04.751943 24389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:04.754102 24389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "454af9d575f742dc8f9c56b26d5c11f3"
format_stamp: "Formatted at 2026-08-08 04:35:04 on dist-test-slave-drl0"
server_key: "532443b1c873f9980cae0bb50777b6da"
server_key_iv: "8d74fee4f016c861804c97aab9dbd5c3"
server_key_version: "encryptionkey@0"
I20260808 04:35:04.754624 24389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "454af9d575f742dc8f9c56b26d5c11f3"
format_stamp: "Formatted at 2026-08-08 04:35:04 on dist-test-slave-drl0"
server_key: "532443b1c873f9980cae0bb50777b6da"
server_key_iv: "8d74fee4f016c861804c97aab9dbd5c3"
server_key_version: "encryptionkey@0"
I20260808 04:35:04.758584 24389 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 04:35:04.761457 24416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:04.763110 24389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:35:04.763271 24389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "454af9d575f742dc8f9c56b26d5c11f3"
format_stamp: "Formatted at 2026-08-08 04:35:04 on dist-test-slave-drl0"
server_key: "532443b1c873f9980cae0bb50777b6da"
server_key_iv: "8d74fee4f016c861804c97aab9dbd5c3"
server_key_version: "encryptionkey@0"
I20260808 04:35:04.763398 24389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:04.787791 24389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:04.791415 24389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:04.791630 24389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:04.792308 24389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:04.793200 24389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:04.793277 24389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:04.793345 24389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:04.793390 24389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:04.806118 24389 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:32967
I20260808 04:35:04.806304 24573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:32967 every 8 connection(s)
I20260808 04:35:04.807298 24389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:35:04.812489 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 24389
I20260808 04:35:04.812618 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:35:04.812976 16273 external_mini_cluster.cc:1428] Setting key 790e699be259d3b22684219f2d5d9cf0
Aug 08 04:35:04 dist-test-slave-drl0 krb5kdc[24238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163704, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:35:04.825800 24355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:04.809947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:51867 (local address 127.15.228.126:42011)
0808 04:35:04.810180 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:04.810184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:04.811495 (+ 1311us) server_negotiation.cc:407] Connection header received
0808 04:35:04.813206 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:04.813210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:04.813292 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:04.813406 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:04.814867 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:04.815822 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:04.817038 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:04.817405 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:04.817596 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:04.820891 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:04.820896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:04.820900 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:04.820939 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:04.823272 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:04.823867 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:04.823871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:04.823874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:04.823976 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:04.824405 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:04.824410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:04.824415 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:04.824704 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:04.824844 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:04.825377 (+ 533us) server_negotiation.cc:299] Negotiation successful
0808 04:35:04.825546 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
WARNING: no policy specified for kudu/127.15.228.66@KRBTEST.COM; defaulting to no policy
I20260808 04:35:04.826545 24578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:04.810546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:42011 (local address 127.15.228.65:51867)
0808 04:35:04.811276 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:04.811342 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:04.812853 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:04.813596 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:04.813604 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:04.814066 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:04.814567 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:04.814606 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:04.815988 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:04.815991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:04.816900 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:04.816911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:04.817849 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:04.817856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:04.818120 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:04.818866 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:04.818894 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:04.820684 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:04.823434 (+ 2750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:04.823440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:04.823454 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:04.823741 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:04.824113 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:04.824116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:04.824117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:04.824283 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:04.824848 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:04.824855 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:04.825104 (+ 249us) client_negotiation.cc:769] Sending connection context
0808 04:35:04.825274 (+ 170us) client_negotiation.cc:240] Negotiation successful
0808 04:35:04.825563 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":141,"threads_started":1}
I20260808 04:35:04.828403 24574 heartbeater.cc:347] Connected to a master server at 127.15.228.126:42011
I20260808 04:35:04.828814 24574 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.66@KRBTEST.COM" created.
I20260808 04:35:04.829706 24574 heartbeater.cc:511] Master 127.15.228.126:42011 requested a full tablet report, sending...
I20260808 04:35:04.831593 24301 ts_manager.cc:194] Registered new tserver with Master: 454af9d575f742dc8f9c56b26d5c11f3 (127.15.228.65:32967)
I20260808 04:35:04.832787 24301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:51867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
I20260808 04:35:04.848946 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:42011
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:43719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:04.992625 24581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:04.992887 24581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:04.992935 24581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:04.996990 24581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:04.997056 24581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:04.997115 24581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:35:05.003382 24581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:43719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:42011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:05.004716 24581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:05.005831 24581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:05.018105 24595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:05.018276 24588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:05.018314 24589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:05.018373 24581 server_base.cc:1034] running on GCE node
I20260808 04:35:05.019284 24581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:05.020018 24581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:05.021237 24581 hybrid_clock.cc:648] HybridClock initialized: now 1786163705021199 us; error 57 us; skew 500 ppm
Aug 08 04:35:05 dist-test-slave-drl0 krb5kdc[24238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163705, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:05.024722 24581 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:35:05.026083 24581 webserver.cc:460] Webserver started at http://127.15.228.66:46833/ using document root <none> and password file <none>
I20260808 04:35:05.026889 24581 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:05.027117 24581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:05.027684 24581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:05.030143 24581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "0b6f2c8db74341f8bee3c9633334ddd5"
format_stamp: "Formatted at 2026-08-08 04:35:05 on dist-test-slave-drl0"
server_key: "44fe98576ec7c9e8b2ef4399dde7a18a"
server_key_iv: "f5434a4a0f2ba915db82e62fde6c2d50"
server_key_version: "encryptionkey@0"
I20260808 04:35:05.030743 24581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "0b6f2c8db74341f8bee3c9633334ddd5"
format_stamp: "Formatted at 2026-08-08 04:35:05 on dist-test-slave-drl0"
server_key: "44fe98576ec7c9e8b2ef4399dde7a18a"
server_key_iv: "f5434a4a0f2ba915db82e62fde6c2d50"
server_key_version: "encryptionkey@0"
I20260808 04:35:05.034842 24581 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:35:05.037792 24603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:05.039678 24581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:35:05.039851 24581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "0b6f2c8db74341f8bee3c9633334ddd5"
format_stamp: "Formatted at 2026-08-08 04:35:05 on dist-test-slave-drl0"
server_key: "44fe98576ec7c9e8b2ef4399dde7a18a"
server_key_iv: "f5434a4a0f2ba915db82e62fde6c2d50"
server_key_version: "encryptionkey@0"
I20260808 04:35:05.039980 24581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:05.060972 24581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:05.064504 24581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:05.064738 24581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:05.065374 24581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:05.066249 24581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:05.066327 24581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:05.066417 24581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:05.066460 24581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:05.076787 24581 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:33303
I20260808 04:35:05.076874 24760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:33303 every 8 connection(s)
I20260808 04:35:05.077891 24581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:35:05.079375 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 24581
I20260808 04:35:05.079504 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:35:05.079841 16273 external_mini_cluster.cc:1428] Setting key 6ed4b27d44ede3c298c569b3f7cd8ba0
Aug 08 04:35:05 dist-test-slave-drl0 krb5kdc[24238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163705, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.228.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.67@KRBTEST.COM" created.
I20260808 04:35:05.094784 24355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:05.080055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:55969 (local address 127.15.228.126:42011)
0808 04:35:05.080225 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:05.080230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:05.081157 (+ 927us) server_negotiation.cc:407] Connection header received
0808 04:35:05.082604 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:05.082607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:05.082664 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:05.082737 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:05.084274 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:05.084821 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:05.086036 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:05.086384 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:05.086509 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:05.089151 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:05.089155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:05.089158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:05.089193 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:05.091481 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:05.092357 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:05.092363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:05.092364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:05.092466 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:05.092855 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:05.092859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:05.092861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:05.093102 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:05.093426 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:05.094438 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0808 04:35:05.094610 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:35:05.096450 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:05.080461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:42011 (local address 127.15.228.66:55969)
0808 04:35:05.081008 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:05.081061 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:05.082390 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:05.082918 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:05.082968 (+ 50us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:05.083345 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:05.084068 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:05.084090 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:05.084958 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:05.084962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:05.085895 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:05.085905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:05.086555 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:05.086559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:05.086749 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:05.087472 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:05.087489 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:05.088964 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:05.091633 (+ 2669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:05.091639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:05.091656 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:05.091922 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:05.092568 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:05.092572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:05.092574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:05.092739 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:05.093426 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:05.093438 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:05.093995 (+ 557us) client_negotiation.cc:769] Sending connection context
0808 04:35:05.094314 (+ 319us) client_negotiation.cc:240] Negotiation successful
0808 04:35:05.094843 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":179,"threads_started":1}
I20260808 04:35:05.098040 24765 heartbeater.cc:347] Connected to a master server at 127.15.228.126:42011
I20260808 04:35:05.098444 24765 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:05.099087 24765 heartbeater.cc:511] Master 127.15.228.126:42011 requested a full tablet report, sending...
I20260808 04:35:05.100348 24301 ts_manager.cc:194] Registered new tserver with Master: 0b6f2c8db74341f8bee3c9633334ddd5 (127.15.228.66:33303)
I20260808 04:35:05.100987 24301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:55969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
I20260808 04:35:05.115396 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:42011
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:43719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:05.248297 24771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:05.248564 24771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:05.248608 24771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:05.252617 24771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:05.252678 24771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:05.252737 24771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:35:05.258021 24771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:43719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:42011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:05.259229 24771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:05.260257 24771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:05.270901 24786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:05.271189 24771 server_base.cc:1034] running on GCE node
W20260808 04:35:05.270991 24782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:05.271143 24783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:05.272193 24771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:05.273029 24771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:05.274256 24771 hybrid_clock.cc:648] HybridClock initialized: now 1786163705274205 us; error 66 us; skew 500 ppm
Aug 08 04:35:05 dist-test-slave-drl0 krb5kdc[24238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163705, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:05.277348 24771 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:35:05.278266 24771 webserver.cc:460] Webserver started at http://127.15.228.67:41147/ using document root <none> and password file <none>
I20260808 04:35:05.278820 24771 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:05.278884 24771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:05.279153 24771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:05.281230 24771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "a7ea7957864d4e2b85baae7b7bff1669"
format_stamp: "Formatted at 2026-08-08 04:35:05 on dist-test-slave-drl0"
server_key: "eea9a135486f5906b8a4d85e0b5fb31c"
server_key_iv: "91e15a623bb21297903c29cf0c4fd5aa"
server_key_version: "encryptionkey@0"
I20260808 04:35:05.281749 24771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "a7ea7957864d4e2b85baae7b7bff1669"
format_stamp: "Formatted at 2026-08-08 04:35:05 on dist-test-slave-drl0"
server_key: "eea9a135486f5906b8a4d85e0b5fb31c"
server_key_iv: "91e15a623bb21297903c29cf0c4fd5aa"
server_key_version: "encryptionkey@0"
I20260808 04:35:05.285732 24771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:35:05.288582 24797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:05.290115 24771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:35:05.290285 24771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "a7ea7957864d4e2b85baae7b7bff1669"
format_stamp: "Formatted at 2026-08-08 04:35:05 on dist-test-slave-drl0"
server_key: "eea9a135486f5906b8a4d85e0b5fb31c"
server_key_iv: "91e15a623bb21297903c29cf0c4fd5aa"
server_key_version: "encryptionkey@0"
I20260808 04:35:05.290411 24771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:05.306424 24771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:05.311575 24771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:05.311858 24771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:05.312690 24771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:05.314105 24771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:05.314226 24771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:05.314324 24771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:05.314384 24771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:05.326231 24771 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:36757
I20260808 04:35:05.326256 24963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:36757 every 8 connection(s)
I20260808 04:35:05.327208 24771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:35:05.335783 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 24771
I20260808 04:35:05.335973 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:35:05.336266 16273 external_mini_cluster.cc:1428] Setting key c4838b1f6245732c928ef27421759936
Aug 08 04:35:05 dist-test-slave-drl0 krb5kdc[24238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163705, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:35:05.346374 24355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:05.329356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:60791 (local address 127.15.228.126:42011)
0808 04:35:05.329562 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:05.329567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:05.330767 (+ 1200us) server_negotiation.cc:407] Connection header received
0808 04:35:05.332718 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:05.332721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:05.332779 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:05.332871 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:05.334762 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:05.335484 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:05.337175 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:05.337504 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:05.337638 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:05.341288 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:05.341292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:05.341294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:05.341325 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:05.343239 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:05.344199 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:05.344202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:05.344204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:05.344290 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:05.344862 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:05.344867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:05.344869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:05.345112 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:05.345239 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:05.346042 (+ 803us) server_negotiation.cc:299] Negotiation successful
0808 04:35:05.346192 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260808 04:35:05.347452 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:05.329762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:42011 (local address 127.15.228.67:60791)
0808 04:35:05.330483 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:05.330552 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:05.332435 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:05.333255 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:05.333266 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:05.333795 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:05.334497 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:05.334516 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:05.335669 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:05.335673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:05.337020 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:05.337033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:05.337730 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:05.337736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:05.337975 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:05.339003 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:05.339027 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:05.341084 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:05.343449 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:05.343500 (+ 51us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:05.343532 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:05.344008 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:05.344457 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:05.344460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:05.344462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:05.344724 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:05.345240 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:05.345249 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:05.345662 (+ 413us) client_negotiation.cc:769] Sending connection context
0808 04:35:05.345999 (+ 337us) client_negotiation.cc:240] Negotiation successful
0808 04:35:05.346409 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":136,"threads_started":1}
I20260808 04:35:05.348852 24966 heartbeater.cc:347] Connected to a master server at 127.15.228.126:42011
I20260808 04:35:05.349079 24966 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:05.349752 24966 heartbeater.cc:511] Master 127.15.228.126:42011 requested a full tablet report, sending...
I20260808 04:35:05.351431 24301 ts_manager.cc:194] Registered new tserver with Master: a7ea7957864d4e2b85baae7b7bff1669 (127.15.228.67:36757)
I20260808 04:35:05.352180 24301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:60791
I20260808 04:35:05.364447 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/test-admin.keytab.
Aug 08 04:35:05 dist-test-slave-drl0 krb5kdc[24238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:05.385229 16273 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 04:35:05.389465 16273 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/krb5cc' of type FILE
W20260808 04:35:05.389542 24976 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:05.387025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51998 (local address 127.15.228.65:32967)
0808 04:35:05.387437 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:05.387446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:05.387469 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:35:05.387520 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:05.387525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:05.387725 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:05.387856 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:05.389209 (+ 1353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51998: BlockingRecv error: recv got EOF from 127.0.0.1:51998 (error 108)
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":154,"threads_started":1}
Aug 08 04:35:05 dist-test-slave-drl0 krb5kdc[24238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:05.390862 16273 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 04:35:05 dist-test-slave-drl0 krb5kdc[24238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:05.392009 16273 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 04:35:05 dist-test-slave-drl0 krb5kdc[24238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.65@KRBTEST.COM
I20260808 04:35:05.405373 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:05.393658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52014 (local address 127.15.228.65:32967)
0808 04:35:05.393846 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:05.393850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:05.393949 (+ 99us) server_negotiation.cc:407] Connection header received
0808 04:35:05.394255 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:05.394259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:05.394341 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:05.394440 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:05.395251 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:05.396076 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:05.396966 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:05.397361 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:05.397501 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:05.400707 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:05.400719 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:05.400735 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:05.400766 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:05.403021 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:05.403601 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:05.403607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:05.403608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:05.403697 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:05.404060 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:05.404063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:05.404064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:05.404403 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:05.404570 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:05.404961 (+ 391us) server_negotiation.cc:299] Negotiation successful
0808 04:35:05.405225 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260808 04:35:05.409483 24976 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:05.408358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52018 (local address 127.15.228.65:32967)
0808 04:35:05.408527 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:05.408532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:05.408632 (+ 100us) server_negotiation.cc:407] Connection header received
0808 04:35:05.408786 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:05.408790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:05.408850 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:05.408931 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:05.409368 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52018: BlockingRecv error: recv got EOF from 127.0.0.1:52018 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260808 04:35:05.409636 16273 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/krb5cc' of type FILE
Aug 08 04:35:05 dist-test-slave-drl0 krb5kdc[24238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:05.410733 16273 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 04:35:05 dist-test-slave-drl0 krb5kdc[24238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:05.411911 16273 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 04:35:05 dist-test-slave-drl0 krb5kdc[24238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.65@KRBTEST.COM
I20260808 04:35:05.422405 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:05.412864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.15.228.65:32967)
0808 04:35:05.412983 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:05.412986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:05.413014 (+ 28us) server_negotiation.cc:407] Connection header received
0808 04:35:05.413105 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:05.413107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:05.413139 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:05.413208 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:05.413789 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:05.414392 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:05.415355 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:05.415589 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:05.415687 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:05.418588 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:05.418591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:05.418593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:05.418617 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:05.420233 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:05.420935 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:05.420938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:05.420940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:05.421038 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:05.421439 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:05.421442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:05.421443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:05.421636 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:05.421773 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:05.422157 (+ 384us) server_negotiation.cc:299] Negotiation successful
0808 04:35:05.422269 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260808 04:35:05.426107 24976 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:05.425029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52040 (local address 127.15.228.65:32967)
0808 04:35:05.425175 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:05.425180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:05.425278 (+ 98us) server_negotiation.cc:407] Connection header received
0808 04:35:05.425444 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:05.425447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:05.425496 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:05.425576 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:05.426009 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52040: BlockingRecv error: recv got EOF from 127.0.0.1:52040 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260808 04:35:05.426239 16273 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786163668922094-16273-0/krb5kdc/krb5cc' of type FILE
Aug 08 04:35:05 dist-test-slave-drl0 krb5kdc[24238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:05.427546 16273 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 04:35:05 dist-test-slave-drl0 krb5kdc[24238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:05.428817 16273 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 04:35:05 dist-test-slave-drl0 krb5kdc[24238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.65@KRBTEST.COM
I20260808 04:35:05.439373 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:05.429741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52044 (local address 127.15.228.65:32967)
0808 04:35:05.429874 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:05.429877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:05.429891 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:35:05.430058 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:05.430061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:05.430099 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:05.430174 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:05.430723 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:05.431304 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:05.432568 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:05.432795 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:05.432909 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:05.435722 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:05.435725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:05.435727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:05.435751 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:05.437212 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:05.437759 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:05.437765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:05.437766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:05.437935 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:05.438411 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:05.438414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:05.438415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:05.438618 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:05.438854 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:05.439067 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 04:35:05.439168 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 04:35:05.440775 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 24389
I20260808 04:35:05.457314 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 24581
I20260808 04:35:05.468168 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 24771
I20260808 04:35:05.476784 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 24254
2026-08-08T04:35:05Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2614 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:35:05 dist-test-slave-drl0 krb5kdc[24991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:35:05 dist-test-slave-drl0 krb5kdc[24991](info): set up 2 sockets
Aug 08 04:35:05 dist-test-slave-drl0 krb5kdc[24991](info): commencing operation
krb5kdc: starting...
W20260808 04:35:06.743167 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:35:06 dist-test-slave-drl0 krb5kdc[24991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:35:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:35:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.126.keytab.
Entry for principal oryx/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.126.keytab.
I20260808 04:35:06.903245 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:36589
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.126.keytab
--principal=oryx/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:36589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:07.026008 25009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:07.026275 25009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:07.026317 25009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:07.032164 25009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:35:07.032249 25009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:07.032272 25009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:07.032291 25009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:35:07.032313 25009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:35:07.040020 25009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:36589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.126.keytab
--principal=oryx/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:36589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:07.041240 25009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:07.042546 25009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:07.052078 25015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:07.052096 25018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:07.052117 25016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:07.052942 25009 server_base.cc:1034] running on GCE node
I20260808 04:35:07.053881 25009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:07.055682 25009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:07.057035 25009 hybrid_clock.cc:648] HybridClock initialized: now 1786163707056963 us; error 65 us; skew 500 ppm
Aug 08 04:35:07 dist-test-slave-drl0 krb5kdc[24991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163707, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:07.064425 25009 init.cc:377] Logged in from keytab as oryx/127.15.228.126@KRBTEST.COM (short username oryx)
I20260808 04:35:07.065466 25009 webserver.cc:460] Webserver started at http://127.15.228.126:33263/ using document root <none> and password file <none>
I20260808 04:35:07.066082 25009 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:07.066177 25009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:07.066426 25009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:07.068928 25009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "7eb94118f70c4552bcab3f523c207610"
format_stamp: "Formatted at 2026-08-08 04:35:07 on dist-test-slave-drl0"
server_key: "6b8918ec7e5bbca34d35139ef222938d"
server_key_iv: "b00120d01455f62e88054c3dc5a1db5b"
server_key_version: "encryptionkey@0"
I20260808 04:35:07.069608 25009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "7eb94118f70c4552bcab3f523c207610"
format_stamp: "Formatted at 2026-08-08 04:35:07 on dist-test-slave-drl0"
server_key: "6b8918ec7e5bbca34d35139ef222938d"
server_key_iv: "b00120d01455f62e88054c3dc5a1db5b"
server_key_version: "encryptionkey@0"
I20260808 04:35:07.074295 25009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260808 04:35:07.077930 25031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:07.079514 25009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:35:07.079684 25009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "7eb94118f70c4552bcab3f523c207610"
format_stamp: "Formatted at 2026-08-08 04:35:07 on dist-test-slave-drl0"
server_key: "6b8918ec7e5bbca34d35139ef222938d"
server_key_iv: "b00120d01455f62e88054c3dc5a1db5b"
server_key_version: "encryptionkey@0"
I20260808 04:35:07.079819 25009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:07.098297 25009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:07.103008 25009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:07.103288 25009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:07.112017 25009 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:36589
I20260808 04:35:07.112344 25103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:36589 every 8 connection(s)
I20260808 04:35:07.113173 25009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:35:07.116596 25104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:07.122397 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 25009
I20260808 04:35:07.122567 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:35:07.123126 16273 external_mini_cluster.cc:1428] Setting key 41a332c654719689671f39b4d808b9a7
I20260808 04:35:07.125103 25104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610: Bootstrap starting.
I20260808 04:35:07.128371 25104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:07.129673 25104 log.cc:824] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:07.132887 25104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610: No bootstrap required, opened a new log
Aug 08 04:35:07 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM
I20260808 04:35:07.138669 25104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } }
I20260808 04:35:07.139034 25104 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:07.139137 25104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb94118f70c4552bcab3f523c207610, State: Initialized, Role: FOLLOWER
I20260808 04:35:07.139700 25104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } }
I20260808 04:35:07.139891 25104 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:35:07.140000 25104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:35:07.140116 25104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:07.141584 25104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } }
I20260808 04:35:07.142040 25104 leader_election.cc:302] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eb94118f70c4552bcab3f523c207610; no voters:
I20260808 04:35:07.142427 25104 leader_election.cc:288] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:35:07.142602 25112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:35:07.142848 25112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb94118f70c4552bcab3f523c207610, State: Running, Role: LEADER
I20260808 04:35:07.143047 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:07.125053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41464 (local address 127.15.228.126:36589)
0808 04:35:07.125653 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:07.125677 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:07.125735 (+ 58us) server_negotiation.cc:407] Connection header received
0808 04:35:07.126690 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:07.126713 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:07.127296 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:07.127866 (+ 570us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:07.129227 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:07.130476 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:07.131788 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:07.132480 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:07.132705 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:07.136697 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:07.136703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:07.136719 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:07.136749 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:07.139383 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:07.139849 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:07.139855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:07.139861 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:07.140011 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:07.140403 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:07.140410 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:07.140412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:07.141010 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:07.141267 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:07.141524 (+ 257us) server_negotiation.cc:299] Negotiation successful
0808 04:35:07.141884 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":172,"threads_started":1}
I20260808 04:35:07.143411 25112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } }
I20260808 04:35:07.143971 25104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:35:07.145486 25113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eb94118f70c4552bcab3f523c207610" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } }
I20260808 04:35:07.145649 25113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:07.145486 25114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eb94118f70c4552bcab3f523c207610. Latest consensus state: current_term: 1 leader_uuid: "7eb94118f70c4552bcab3f523c207610" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } }
I20260808 04:35:07.145965 25114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:07.146246 25126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:35:07.149488 25126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:35:07.156929 25126 catalog_manager.cc:1347] Generated new cluster ID: 7b0fb3629a754d6186ff6c61f948734c
I20260808 04:35:07.157018 25126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:35:07.189013 25126 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:35:07.191115 25126 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:35:07.200121 25126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610: Generated new TSK 0
I20260808 04:35:07.201025 25126 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.65.keytab.
Entry for principal oryx/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.65.keytab.
I20260808 04:35:07.247992 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:36589
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.65.keytab
--principal=oryx/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:07.370721 25145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:07.371029 25145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:07.371074 25145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:07.375041 25145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:07.375097 25145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:07.375154 25145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:35:07.380250 25145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.65.keytab
--principal=oryx/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:36589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:07.381918 25145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:07.383366 25145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:07.395902 25152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:07.395949 25154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:07.395902 25157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:07.396205 25145 server_base.cc:1034] running on GCE node
I20260808 04:35:07.396700 25145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:07.397351 25145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:07.398530 25145 hybrid_clock.cc:648] HybridClock initialized: now 1786163707398491 us; error 84 us; skew 500 ppm
Aug 08 04:35:07 dist-test-slave-drl0 krb5kdc[24991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163707, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:07.402493 25145 init.cc:377] Logged in from keytab as oryx/127.15.228.65@KRBTEST.COM (short username oryx)
I20260808 04:35:07.403575 25145 webserver.cc:460] Webserver started at http://127.15.228.65:39711/ using document root <none> and password file <none>
I20260808 04:35:07.404173 25145 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:07.404242 25145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:07.404469 25145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:07.406862 25145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "0eba8ed56d6047daa77802d8f0ffc71c"
format_stamp: "Formatted at 2026-08-08 04:35:07 on dist-test-slave-drl0"
server_key: "6444f345b9f338e3a580389faa935cf2"
server_key_iv: "d4f4106dc268c3722f92c0130f95f0f6"
server_key_version: "encryptionkey@0"
I20260808 04:35:07.407549 25145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "0eba8ed56d6047daa77802d8f0ffc71c"
format_stamp: "Formatted at 2026-08-08 04:35:07 on dist-test-slave-drl0"
server_key: "6444f345b9f338e3a580389faa935cf2"
server_key_iv: "d4f4106dc268c3722f92c0130f95f0f6"
server_key_version: "encryptionkey@0"
I20260808 04:35:07.411793 25145 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:35:07.414351 25166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:07.415678 25145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260808 04:35:07.415776 25145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "0eba8ed56d6047daa77802d8f0ffc71c"
format_stamp: "Formatted at 2026-08-08 04:35:07 on dist-test-slave-drl0"
server_key: "6444f345b9f338e3a580389faa935cf2"
server_key_iv: "d4f4106dc268c3722f92c0130f95f0f6"
server_key_version: "encryptionkey@0"
I20260808 04:35:07.415910 25145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:07.435395 25145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:07.438855 25145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:07.439143 25145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:07.439792 25145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:07.440867 25145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:07.440971 25145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:07.441054 25145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:07.441102 25145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:07.451588 25145 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:45007
I20260808 04:35:07.451776 25316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:45007 every 8 connection(s)
I20260808 04:35:07.452548 25145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:35:07.456130 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 25145
I20260808 04:35:07.456328 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:35:07.456722 16273 external_mini_cluster.cc:1428] Setting key 4e6ed96f93d912c98faa12b580b976d8
Aug 08 04:35:07 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163707, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.15.228.66@KRBTEST.COM; defaulting to no policy
I20260808 04:35:07.471792 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:07.454733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:36107 (local address 127.15.228.126:36589)
0808 04:35:07.454900 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:07.454904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:07.456343 (+ 1439us) server_negotiation.cc:407] Connection header received
0808 04:35:07.458349 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:07.458353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:07.458412 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:07.458574 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:07.460098 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:07.461000 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:07.463547 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:07.463796 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:07.464073 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:07.467477 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:07.467482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:07.467485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:07.467520 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:07.469341 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:07.469913 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:07.469928 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:07.469933 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:07.470032 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:07.470488 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:07.470491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:07.470492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:07.470710 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:07.470806 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:07.471513 (+ 707us) server_negotiation.cc:299] Negotiation successful
0808 04:35:07.471632 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 04:35:07.472747 25319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:07.455140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.65:36107)
0808 04:35:07.456000 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:07.456108 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:07.457976 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:07.458865 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:07.458876 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:07.459393 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:07.459913 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:07.459928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:07.461242 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:07.461250 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:07.463329 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:07.463346 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:07.464055 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:07.464062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:07.464303 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:07.465221 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:07.465239 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:07.467249 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:07.469498 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:07.469503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:07.469516 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:07.469774 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:07.470158 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:07.470160 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:07.470162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:07.470316 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:07.470898 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:07.470903 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:07.471223 (+ 320us) client_negotiation.cc:769] Sending connection context
0808 04:35:07.471399 (+ 176us) client_negotiation.cc:240] Negotiation successful
0808 04:35:07.471681 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":131,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.228.66@KRBTEST.COM" created.
I20260808 04:35:07.474076 25317 heartbeater.cc:347] Connected to a master server at 127.15.228.126:36589
I20260808 04:35:07.474367 25317 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:07.475121 25317 heartbeater.cc:511] Master 127.15.228.126:36589 requested a full tablet report, sending...
I20260808 04:35:07.476955 25052 ts_manager.cc:194] Registered new tserver with Master: 0eba8ed56d6047daa77802d8f0ffc71c (127.15.228.65:45007)
I20260808 04:35:07.478108 25052 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:36107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.66.keytab.
Entry for principal oryx/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.66.keytab.
I20260808 04:35:07.493973 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:36589
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.66.keytab
--principal=oryx/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:07.629550 25323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:07.629966 25323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:07.630023 25323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:07.635437 25323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:07.635510 25323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:07.635620 25323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:35:07.641397 25323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.66.keytab
--principal=oryx/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:36589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:07.642549 25323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:07.643775 25323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:07.654768 25330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:07.655025 25323 server_base.cc:1034] running on GCE node
W20260808 04:35:07.654786 25334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:07.654768 25329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:07.655597 25323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:07.656272 25323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:07.657581 25323 hybrid_clock.cc:648] HybridClock initialized: now 1786163707657541 us; error 31 us; skew 500 ppm
Aug 08 04:35:07 dist-test-slave-drl0 krb5kdc[24991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163707, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:07.661674 25323 init.cc:377] Logged in from keytab as oryx/127.15.228.66@KRBTEST.COM (short username oryx)
I20260808 04:35:07.662719 25323 webserver.cc:460] Webserver started at http://127.15.228.66:36231/ using document root <none> and password file <none>
I20260808 04:35:07.663420 25323 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:07.663518 25323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:07.663981 25323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:07.667533 25323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "93fc6b49759b480a8cd03c65ecb21bb9"
format_stamp: "Formatted at 2026-08-08 04:35:07 on dist-test-slave-drl0"
server_key: "1c45af87e145d521a15c4ba19858405f"
server_key_iv: "104389daac7ab9bce2ed76f530e5a772"
server_key_version: "encryptionkey@0"
I20260808 04:35:07.668628 25323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "93fc6b49759b480a8cd03c65ecb21bb9"
format_stamp: "Formatted at 2026-08-08 04:35:07 on dist-test-slave-drl0"
server_key: "1c45af87e145d521a15c4ba19858405f"
server_key_iv: "104389daac7ab9bce2ed76f530e5a772"
server_key_version: "encryptionkey@0"
I20260808 04:35:07.673305 25323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:35:07.676180 25346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:07.677652 25323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:35:07.677827 25323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "93fc6b49759b480a8cd03c65ecb21bb9"
format_stamp: "Formatted at 2026-08-08 04:35:07 on dist-test-slave-drl0"
server_key: "1c45af87e145d521a15c4ba19858405f"
server_key_iv: "104389daac7ab9bce2ed76f530e5a772"
server_key_version: "encryptionkey@0"
I20260808 04:35:07.677971 25323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:07.697214 25323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:07.701206 25323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:07.701458 25323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:07.702095 25323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:07.703058 25323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:07.703127 25323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:07.703197 25323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:07.703248 25323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:07.715991 25323 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:34177
I20260808 04:35:07.716027 25495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:34177 every 8 connection(s)
I20260808 04:35:07.716956 25323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:35:07.724927 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 25323
I20260808 04:35:07.725028 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:35:07.725320 16273 external_mini_cluster.cc:1428] Setting key 366f85adcb6fff0b8b76618bb2726a75
Aug 08 04:35:07 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163707, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM
I20260808 04:35:07.733040 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:07.719097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:52687 (local address 127.15.228.126:36589)
0808 04:35:07.719252 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:07.719256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:07.720144 (+ 888us) server_negotiation.cc:407] Connection header received
0808 04:35:07.721655 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:07.721659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:07.721916 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:07.722183 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:07.723460 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:07.724029 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:07.725267 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:07.725623 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:07.725759 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:07.728553 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:07.728557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:07.728560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:07.728594 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:07.730112 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:07.730680 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:07.730683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:07.730685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:07.730768 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:07.731278 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:07.731281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:07.731292 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:07.731504 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:07.731598 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:07.732475 (+ 877us) server_negotiation.cc:299] Negotiation successful
0808 04:35:07.732781 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:35:07.734411 25498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:07.719496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.66:52687)
0808 04:35:07.719997 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:07.720039 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:07.721446 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:07.722349 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:07.722356 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:07.722721 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:07.723292 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:07.723305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:07.724168 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:07.724171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:07.725141 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:07.725151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:07.725808 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:07.725812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:07.726046 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:07.726737 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:07.726754 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:07.728379 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:07.730275 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:07.730281 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:07.730297 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:07.730563 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:07.730885 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:07.730892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:07.730896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:07.731112 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:07.731625 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:07.731635 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:07.731976 (+ 341us) client_negotiation.cc:769] Sending connection context
0808 04:35:07.732188 (+ 212us) client_negotiation.cc:240] Negotiation successful
0808 04:35:07.732618 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":117,"threads_started":1}
I20260808 04:35:07.735983 25496 heartbeater.cc:347] Connected to a master server at 127.15.228.126:36589
I20260808 04:35:07.736258 25496 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:07.736927 25496 heartbeater.cc:511] Master 127.15.228.126:36589 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.15.228.67@KRBTEST.COM; defaulting to no policy
I20260808 04:35:07.738360 25052 ts_manager.cc:194] Registered new tserver with Master: 93fc6b49759b480a8cd03c65ecb21bb9 (127.15.228.66:34177)
I20260808 04:35:07.739073 25052 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:52687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.228.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.67.keytab.
Entry for principal oryx/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.67.keytab.
I20260808 04:35:07.763422 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:36589
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.67.keytab
--principal=oryx/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:07.890079 25506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:07.890331 25506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:07.890373 25506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:07.894673 25506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:07.894743 25506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:07.894805 25506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:35:07.900177 25506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.67.keytab
--principal=oryx/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:36589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:07.901193 25506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:07.902242 25506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:07.912698 25514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:07.912809 25506 server_base.cc:1034] running on GCE node
W20260808 04:35:07.912719 25516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:07.912698 25513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:07.913426 25506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:07.914150 25506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:07.915390 25506 hybrid_clock.cc:648] HybridClock initialized: now 1786163707915369 us; error 56 us; skew 500 ppm
Aug 08 04:35:07 dist-test-slave-drl0 krb5kdc[24991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163707, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:07.919370 25506 init.cc:377] Logged in from keytab as oryx/127.15.228.67@KRBTEST.COM (short username oryx)
I20260808 04:35:07.920907 25506 webserver.cc:460] Webserver started at http://127.15.228.67:35127/ using document root <none> and password file <none>
I20260808 04:35:07.921758 25506 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:07.921873 25506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:07.922266 25506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:07.924805 25506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "2114de6f886e4314b4d47f30c9cea294"
format_stamp: "Formatted at 2026-08-08 04:35:07 on dist-test-slave-drl0"
server_key: "3e7193f53e777b84c16a81447b3c88dd"
server_key_iv: "8333bccb76b4642955be28793fc423fb"
server_key_version: "encryptionkey@0"
I20260808 04:35:07.925421 25506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "2114de6f886e4314b4d47f30c9cea294"
format_stamp: "Formatted at 2026-08-08 04:35:07 on dist-test-slave-drl0"
server_key: "3e7193f53e777b84c16a81447b3c88dd"
server_key_iv: "8333bccb76b4642955be28793fc423fb"
server_key_version: "encryptionkey@0"
I20260808 04:35:07.929888 25506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260808 04:35:07.932512 25526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:07.933976 25506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:35:07.934136 25506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "2114de6f886e4314b4d47f30c9cea294"
format_stamp: "Formatted at 2026-08-08 04:35:07 on dist-test-slave-drl0"
server_key: "3e7193f53e777b84c16a81447b3c88dd"
server_key_iv: "8333bccb76b4642955be28793fc423fb"
server_key_version: "encryptionkey@0"
I20260808 04:35:07.934264 25506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:07.946832 25506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:07.951426 25506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:07.951665 25506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:07.952338 25506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:07.953260 25506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:07.953338 25506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:07.953418 25506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:07.953469 25506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:07.965131 25506 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:38279
I20260808 04:35:07.965157 25691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:38279 every 8 connection(s)
I20260808 04:35:07.966136 25506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:35:07.973063 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 25506
I20260808 04:35:07.973202 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:35:07.973520 16273 external_mini_cluster.cc:1428] Setting key 145bb9df145d51aeeb40ab6e5116a2f7
Aug 08 04:35:07 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163707, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM
I20260808 04:35:07.984174 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:07.968894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:43671 (local address 127.15.228.126:36589)
0808 04:35:07.969095 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:07.969100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:07.969943 (+ 843us) server_negotiation.cc:407] Connection header received
0808 04:35:07.971491 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:07.971494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:07.971546 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:07.971633 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:07.972841 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:07.973579 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:07.975652 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:07.976093 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:07.976223 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:07.979682 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:07.979685 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:07.979688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:07.979715 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:07.981312 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:07.982239 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:07.982243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:07.982244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:07.982336 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:07.982872 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:07.982875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:07.982877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:07.983153 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:07.983242 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:07.983847 (+ 605us) server_negotiation.cc:299] Negotiation successful
0808 04:35:07.984011 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 04:35:07.985000 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:07.969261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.67:43671)
0808 04:35:07.969778 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:07.969822 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:07.971261 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:07.971798 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:07.971805 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:07.972171 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:07.972652 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:07.972665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:07.973732 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:07.973736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:07.975396 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:07.975408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:07.976404 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:07.976408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:07.976563 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:07.977386 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:07.977410 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:07.979432 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:07.981492 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:07.981581 (+ 89us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:07.981601 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:07.982072 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:07.982474 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:07.982477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:07.982479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:07.982650 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:07.983270 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:07.983276 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:07.983523 (+ 247us) client_negotiation.cc:769] Sending connection context
0808 04:35:07.983695 (+ 172us) client_negotiation.cc:240] Negotiation successful
0808 04:35:07.983988 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":135,"threads_started":1}
I20260808 04:35:07.987408 25692 heartbeater.cc:347] Connected to a master server at 127.15.228.126:36589
I20260808 04:35:07.987709 25692 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:07.988314 25692 heartbeater.cc:511] Master 127.15.228.126:36589 requested a full tablet report, sending...
I20260808 04:35:07.989473 25052 ts_manager.cc:194] Registered new tserver with Master: 2114de6f886e4314b4d47f30c9cea294 (127.15.228.67:38279)
I20260808 04:35:07.990065 25052 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:43671
I20260808 04:35:08.002985 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.228.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.228.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.228.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.125.keytab.
Entry for principal oryx/127.15.228.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.125.keytab.
I20260808 04:35:08.039613 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.125:42221
--webserver_interface=127.15.228.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.125.keytab
--principal=oryx/127.15.228.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:36589,127.15.228.125:42221
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:08.167155 25697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:08.167511 25697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:08.167611 25697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:08.173200 25697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:35:08.173285 25697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:08.173309 25697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:08.173329 25697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:35:08.173346 25697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:35:08.179169 25697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:36589,127.15.228.125:42221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.125.keytab
--principal=oryx/127.15.228.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.125:42221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.228.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
W20260808 04:35:08.179425 25697 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.15.228.126:36589,127.15.228.125:42221'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 04:35:08.180505 25697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:08.181967 25697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:08.192625 25708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:08.192716 25707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:08.192880 25710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:08.192844 25697 server_base.cc:1034] running on GCE node
I20260808 04:35:08.193797 25697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:08.195013 25697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:08.196303 25697 hybrid_clock.cc:648] HybridClock initialized: now 1786163708196263 us; error 34 us; skew 500 ppm
Aug 08 04:35:08 dist-test-slave-drl0 krb5kdc[24991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163708, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:08.200040 25697 init.cc:377] Logged in from keytab as oryx/127.15.228.125@KRBTEST.COM (short username oryx)
I20260808 04:35:08.201647 25697 webserver.cc:460] Webserver started at http://127.15.228.125:36109/ using document root <none> and password file <none>
I20260808 04:35:08.202486 25697 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:08.202569 25697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:08.203020 25697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:08.205322 25697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/data/instance:
uuid: "fd4721373d7b4a7686b043b6116850eb"
format_stamp: "Formatted at 2026-08-08 04:35:08 on dist-test-slave-drl0"
server_key: "60e3854d5e2da3ba6b67fc0a4a282c58"
server_key_iv: "b441fbecef0eadece3514df75087c158"
server_key_version: "encryptionkey@0"
I20260808 04:35:08.205914 25697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/wal/instance:
uuid: "fd4721373d7b4a7686b043b6116850eb"
format_stamp: "Formatted at 2026-08-08 04:35:08 on dist-test-slave-drl0"
server_key: "60e3854d5e2da3ba6b67fc0a4a282c58"
server_key_iv: "b441fbecef0eadece3514df75087c158"
server_key_version: "encryptionkey@0"
I20260808 04:35:08.210692 25697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260808 04:35:08.213426 25720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:08.215493 25697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:35:08.215677 25697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/wal
uuid: "fd4721373d7b4a7686b043b6116850eb"
format_stamp: "Formatted at 2026-08-08 04:35:08 on dist-test-slave-drl0"
server_key: "60e3854d5e2da3ba6b67fc0a4a282c58"
server_key_iv: "b441fbecef0eadece3514df75087c158"
server_key_version: "encryptionkey@0"
I20260808 04:35:08.215804 25697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:08.234449 25697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:08.237949 25697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:08.238268 25697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:08.246671 25697 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.125:42221
I20260808 04:35:08.247145 25795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.125:42221 every 8 connection(s)
I20260808 04:35:08.247839 25697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/data/info.pb
I20260808 04:35:08.248941 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 25697
I20260808 04:35:08.251784 25797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:08.257169 25797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:35:08 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.228.125@KRBTEST.COM
I20260808 04:35:08.266547 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.250908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34450 (local address 127.15.228.125:42221)
0808 04:35:08.251409 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:08.251435 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:08.251496 (+ 61us) server_negotiation.cc:407] Connection header received
0808 04:35:08.252268 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:08.252285 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:08.252599 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:08.252875 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:08.253824 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.254885 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.256216 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.256720 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.256917 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.260306 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:08.260313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:08.260331 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:08.260375 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:08.262759 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.263582 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.263587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.263592 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.263710 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.264054 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.264057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.264058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.264585 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:08.264854 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:08.265228 (+ 374us) server_negotiation.cc:299] Negotiation successful
0808 04:35:08.265738 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":172,"threads_started":1}
Aug 08 04:35:08 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163708, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.125@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM
I20260808 04:35:08.271060 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.260111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41466 (local address 127.15.228.126:36589)
0808 04:35:08.260281 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:08.260285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:08.260712 (+ 427us) server_negotiation.cc:407] Connection header received
0808 04:35:08.261187 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:08.261193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:08.261262 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:08.261378 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:08.262309 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.263133 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.264303 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.264626 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.264731 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.267316 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:08.267327 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:08.267330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:08.267454 (+ 124us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:08.269193 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.269679 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.269682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.269684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.269770 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.270062 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.270067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.270069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.270299 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:08.270419 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:08.270748 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 04:35:08.270872 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 04:35:08.271080 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.259938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.0.0.1:41466)
0808 04:35:08.260530 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:08.260585 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:08.261015 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:08.261504 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:08.261515 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:08.262017 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:08.262156 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.262162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.263251 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.263256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.264174 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.264184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.264836 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.264840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.264993 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.265672 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:08.265684 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:08.267171 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:08.269313 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.269325 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.269332 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.269562 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.269876 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.269879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.269880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.269966 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.270419 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:08.270427 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:08.270541 (+ 114us) client_negotiation.cc:769] Sending connection context
0808 04:35:08.270686 (+ 145us) client_negotiation.cc:240] Negotiation successful
0808 04:35:08.270874 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":179,"threads_started":1}
I20260808 04:35:08.272766 25797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:35:08 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163708, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.125@KRBTEST.COM for oryx/127.15.228.125@KRBTEST.COM
I20260808 04:35:08.283830 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.273756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:42221 (local address 127.0.0.1:34462)
0808 04:35:08.273907 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:08.273924 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:08.274030 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:08.274370 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:08.274373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:08.274520 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:08.274621 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.274626 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.276237 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.276241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.276820 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.276828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.277426 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.277430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.277515 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.277976 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:08.277988 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:08.279898 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:08.282112 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.282115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.282116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.282337 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.282735 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.282737 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.282739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.282824 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.283388 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:08.283390 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:08.283473 (+ 83us) client_negotiation.cc:769] Sending connection context
0808 04:35:08.283587 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 04:35:08.283699 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 04:35:08.283946 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.273953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.15.228.125:42221)
0808 04:35:08.274138 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:08.274141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:08.274156 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:35:08.274190 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:08.274192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:08.274229 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:08.274308 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:08.274793 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.276020 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.276979 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.277228 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.277324 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.280075 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:08.280078 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:08.280080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:08.280101 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:08.281856 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.282493 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.282497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.282498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.282606 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.283013 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.283017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.283018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.283236 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:08.283341 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:08.283667 (+ 326us) server_negotiation.cc:299] Negotiation successful
0808 04:35:08.283777 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 04:35:08.287472 25797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb: Bootstrap starting.
I20260808 04:35:08.289870 25797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:08.290674 25797 log.cc:824] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:08.292905 25797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb: No bootstrap required, opened a new log
I20260808 04:35:08.296447 25797 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } }
I20260808 04:35:08.296654 25797 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:08.296689 25797 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd4721373d7b4a7686b043b6116850eb, State: Initialized, Role: FOLLOWER
I20260808 04:35:08.297145 25797 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } }
I20260808 04:35:08.297688 25808 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } } }
I20260808 04:35:08.297818 25808 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:35:08.298509 25797 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [sys.catalog]: configured and running, proceeding with master startup.
W20260808 04:35:08.304597 25819 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:35:08.304695 25819 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:35:08.307194 25697 master_runner.cc:428] Detected that this master fd4721373d7b4a7686b043b6116850eb is joining an existing cluster
I20260808 04:35:08.307256 25697 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.228.125:42221
I20260808 04:35:08.317471 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.309882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34464 (local address 127.15.228.125:42221)
0808 04:35:08.309978 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:08.309983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:08.310032 (+ 49us) server_negotiation.cc:407] Connection header received
0808 04:35:08.310163 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:08.310166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:08.310201 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:08.310249 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:08.310899 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.311435 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.312255 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.312466 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.312577 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.313719 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:08.313722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:08.313725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:08.313755 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:08.315566 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.316078 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.316081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.316083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.316163 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.316488 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.316491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.316493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.316713 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:08.316822 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:08.317129 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 04:35:08.317326 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":43}
I20260808 04:35:08.317485 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.309491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:42221 (local address 127.0.0.1:34464)
0808 04:35:08.309959 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:08.309973 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:08.310074 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:08.310389 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:08.310392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:08.310648 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:08.310784 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.310791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.311555 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.311558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.312131 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.312138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.312586 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.312590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.312677 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.313211 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:08.313227 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:08.313582 (+ 355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:08.315697 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.315702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.315704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.315984 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.316259 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.316264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.316266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.316368 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.316844 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:08.316846 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:08.316938 (+ 92us) client_negotiation.cc:769] Sending connection context
0808 04:35:08.317078 (+ 140us) client_negotiation.cc:240] Negotiation successful
0808 04:35:08.317273 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260808 04:35:08.319808 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.309495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.15.228.126:36589)
0808 04:35:08.309631 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:08.309636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:08.309979 (+ 343us) server_negotiation.cc:407] Connection header received
0808 04:35:08.310154 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:08.310157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:08.310203 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:08.310289 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:08.311015 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.311834 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.313056 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.313390 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.313469 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.315476 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:08.315480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:08.315483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:08.315515 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:08.317526 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.318235 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.318239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.318241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.318337 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.318798 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.318802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.318803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.319071 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:08.319155 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:08.319533 (+ 378us) server_negotiation.cc:299] Negotiation successful
0808 04:35:08.319661 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:35:08.320246 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.309446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.0.0.1:41476)
0808 04:35:08.309891 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:08.309907 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:08.310055 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:08.310335 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:08.310340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:08.310668 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:08.310835 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.310844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.311983 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.311986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.312910 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.312921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.313629 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.313632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.313728 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.314846 (+ 1118us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:08.314861 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:08.315313 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:08.317858 (+ 2545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.317861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.317863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.318104 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.318450 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.318453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.318455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.318546 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.319168 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:08.319171 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:08.319272 (+ 101us) client_negotiation.cc:769] Sending connection context
0808 04:35:08.319384 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 04:35:08.319537 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"spinlock_wait_cycles":2304,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for oryx/127.15.228.124@KRBTEST.COM; defaulting to no policy
Aug 08 04:35:08 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163707, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.126@KRBTEST.COM for oryx/127.15.228.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.228.124@KRBTEST.COM" created.
I20260808 04:35:08.337013 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.324985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:42221 (local address 127.0.0.1:34468)
0808 04:35:08.327401 (+ 2416us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:08.327443 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:08.327700 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:08.328038 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:08.328044 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:08.328406 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:08.328516 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.328522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.329246 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.329251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.329994 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.330004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.330506 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.330510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.330634 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.331221 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:08.331234 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:08.332715 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:08.334994 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.335008 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.335069 (+ 61us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.335425 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.335746 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.335749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.335751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.335840 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.336261 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:08.336270 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:08.336404 (+ 134us) client_negotiation.cc:769] Sending connection context
0808 04:35:08.336550 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 04:35:08.336749 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2133,"thread_start_us":89,"threads_started":1}
I20260808 04:35:08.337013 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.324992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.15.228.125:42221)
0808 04:35:08.325111 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:08.325116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:08.327610 (+ 2494us) server_negotiation.cc:407] Connection header received
0808 04:35:08.327844 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:08.327847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:08.327913 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:08.328001 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:08.328633 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.329133 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.330140 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.330380 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.330490 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.332869 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:08.332873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:08.332875 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:08.332903 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:08.334587 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.335544 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.335550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.335551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.335639 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.335944 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.335947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.335949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.336136 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:08.336260 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:08.336623 (+ 363us) server_negotiation.cc:299] Negotiation successful
0808 04:35:08.336744 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 04:35:08.339159 25052 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.228.125:42221: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fd4721373d7b4a7686b043b6116850eb"
member_type: NON_VOTER
last_known_addr {
host: "127.15.228.125"
port: 42221
}
attrs {
promote: true
}
}
dest_uuid: "7eb94118f70c4552bcab3f523c207610"
cas_config_opid_index: -1
I20260808 04:35:08.340253 25052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: NON_VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: true } }
I20260808 04:35:08.343295 25839 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fd4721373d7b4a7686b043b6116850eb (127.15.228.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: NON_VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: true } } }
I20260808 04:35:08.344355 25759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:08.345889 25839 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.228.125:42221
I20260808 04:35:08.345955 25759 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 1 FOLLOWER]: Refusing update from remote peer 7eb94118f70c4552bcab3f523c207610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 04:35:08.346441 25839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7eb94118f70c4552bcab3f523c207610" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: NON_VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: true } } }
I20260808 04:35:08.346848 25697 master.cc:537] Master@127.15.228.125:42221 shutting down...
I20260808 04:35:08.346693 25839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:08.347186 25838 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: NON_VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 04:35:08.350647 25033 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260808 04:35:08.352517 25697 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:35:08.352852 25697 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:08.352977 25697 tablet_replica.cc:330] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.228.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.124.keytab.
Entry for principal oryx/127.15.228.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.124.keytab.
I20260808 04:35:08.354632 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.124:36829
--webserver_interface=127.15.228.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.124.keytab
--principal=oryx/127.15.228.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:36589,127.15.228.125:42221,127.15.228.124:36829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:35:08.360216 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.359667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:42221 (local address 127.0.0.1:34474)
0808 04:35:08.359947 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:08.360076 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:42221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260808 04:35:08.360599 25033 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 -> Peer fd4721373d7b4a7686b043b6116850eb (127.15.228.125:42221): Couldn't send request to peer fd4721373d7b4a7686b043b6116850eb. Status: Network error: Client connection negotiation failed: client connection to 127.15.228.125:42221: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 04:35:08.360976 25697 master.cc:552] Master@127.15.228.125:42221 shutdown complete.
I20260808 04:35:08.361053 25697 master_runner.cc:305] Clearing existing system tablet
I20260808 04:35:08.361326 25697 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:08.361407 25697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:08.362615 25697 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 04:35:08.363641 25846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:08.363879 25697 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:08.364053 25697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 04:35:08.364095 25697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/wal
uuid: "fd4721373d7b4a7686b043b6116850eb"
format_stamp: "Formatted at 2026-08-08 04:35:08 on dist-test-slave-drl0"
server_key: "60e3854d5e2da3ba6b67fc0a4a282c58"
server_key_iv: "b441fbecef0eadece3514df75087c158"
server_key_version: "encryptionkey@0"
I20260808 04:35:08.366053 25697 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:08.368266 25697 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 04:35:08.368391 25697 log.cc:1196] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 04:35:08.368665 25697 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb: Deleting consensus metadata
I20260808 04:35:08.368896 25697 master_runner.cc:315] Copying system tablet from 127.15.228.126:36589
I20260808 04:35:08.370388 25697 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb: tablet copy: Beginning tablet copy session from remote peer at address 127.15.228.126:36589
I20260808 04:35:08.379936 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.371563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41492 (local address 127.15.228.126:36589)
0808 04:35:08.371710 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:08.371715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:08.371730 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:35:08.371785 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:08.371788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:08.371830 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:08.371934 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:08.372435 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.373026 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.373955 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.374216 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.374343 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.375599 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:08.375603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:08.375606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:08.375633 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:08.377316 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.377779 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.377783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.377785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.377868 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.378815 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.378822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.378824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.379168 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:08.379331 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:08.379632 (+ 301us) server_negotiation.cc:299] Negotiation successful
0808 04:35:08.379756 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:35:08.379951 25854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.371188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.0.0.1:41492)
0808 04:35:08.371477 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:08.371494 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:08.371615 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:08.371958 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:08.371961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:08.372185 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:08.372292 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.372299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.373162 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.373168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.373808 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.373817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.374348 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.374352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.374449 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.375072 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:08.375085 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:08.375442 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:08.377456 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.377460 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.377462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.377668 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.377967 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.377971 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.377973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.378180 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.379317 (+ 1137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:08.379320 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:08.379434 (+ 114us) client_negotiation.cc:769] Sending connection context
0808 04:35:08.379556 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 04:35:08.379719 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":124,"threads_started":1}
I20260808 04:35:08.381052 25082 tablet_copy_service.cc:140] P 7eb94118f70c4552bcab3f523c207610: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fd4721373d7b4a7686b043b6116850eb ({username='oryx', principal='oryx/127.15.228.125@KRBTEST.COM'} at 127.0.0.1:41492)
I20260808 04:35:08.381274 25082 tablet_copy_service.cc:161] P 7eb94118f70c4552bcab3f523c207610: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fd4721373d7b4a7686b043b6116850eb at {username='oryx', principal='oryx/127.15.228.125@KRBTEST.COM'} at 127.0.0.1:41492: session id = fd4721373d7b4a7686b043b6116850eb-00000000000000000000000000000000
I20260808 04:35:08.384256 25082 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 04:35:08.386706 25697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 04:35:08.389560 25697 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb: tablet copy: Starting download of 0 data blocks...
I20260808 04:35:08.389802 25697 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb: tablet copy: Starting download of 1 WAL segments...
I20260808 04:35:08.391817 25697 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 04:35:08.393635 25082 tablet_copy_service.cc:342] P 7eb94118f70c4552bcab3f523c207610: Request end of tablet copy session fd4721373d7b4a7686b043b6116850eb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.228.125@KRBTEST.COM'} at 127.0.0.1:41492
I20260808 04:35:08.393774 25082 tablet_copy_service.cc:433] P 7eb94118f70c4552bcab3f523c207610: ending tablet copy session fd4721373d7b4a7686b043b6116850eb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fd4721373d7b4a7686b043b6116850eb
I20260808 04:35:08.401971 25697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:08.403698 25859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:08.404116 25857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:08.404131 25697 server_base.cc:1034] running on GCE node
W20260808 04:35:08.407101 25855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:08.407598 25697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:08.408169 25697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:08.409372 25697 hybrid_clock.cc:648] HybridClock initialized: now 1786163708409351 us; error 54 us; skew 500 ppm
Aug 08 04:35:08 dist-test-slave-drl0 krb5kdc[24991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163708, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:08.411518 25697 init.cc:377] Logged in from keytab as oryx/127.15.228.125@KRBTEST.COM (short username oryx)
I20260808 04:35:08.412240 25697 webserver.cc:460] Webserver started at http://127.15.228.125:34611/ using document root <none> and password file <none>
I20260808 04:35:08.412504 25697 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:08.412566 25697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:08.414003 25697 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 04:35:08.415503 25870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:08.415853 25697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 04:35:08.415962 25697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/wal
uuid: "fd4721373d7b4a7686b043b6116850eb"
format_stamp: "Formatted at 2026-08-08 04:35:08 on dist-test-slave-drl0"
server_key: "60e3854d5e2da3ba6b67fc0a4a282c58"
server_key_iv: "b441fbecef0eadece3514df75087c158"
server_key_version: "encryptionkey@0"
I20260808 04:35:08.416086 25697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:08.440935 25697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:08.443622 25697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:08.450871 25697 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.125:42221
I20260808 04:35:08.451292 25939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.125:42221 every 8 connection(s)
I20260808 04:35:08.451660 25697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-1/data/info.pb
I20260808 04:35:08.453271 25941 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:35:08.454154 25941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb: Bootstrap starting.
I20260808 04:35:08.466166 25941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:35:08.466537 25941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb: Bootstrap complete.
I20260808 04:35:08.467280 25941 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: NON_VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: true } }
I20260808 04:35:08.467450 25941 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fd4721373d7b4a7686b043b6116850eb, State: Initialized, Role: LEARNER
I20260808 04:35:08.467592 25941 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: NON_VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: true } }
I20260808 04:35:08.468007 25944 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: NON_VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: true } } }
I20260808 04:35:08.468290 25944 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [sys.catalog]: This master's current role is: LEARNER
I20260808 04:35:08.468149 25941 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:35:08.472765 25956 catalog_manager.cc:1259] Loaded cluster ID: 7b0fb3629a754d6186ff6c61f948734c
I20260808 04:35:08.472882 25956 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb: loading cluster ID for follower catalog manager: success
I20260808 04:35:08.475160 25956 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb: acquiring CA information for follower catalog manager: success
I20260808 04:35:08.476547 25956 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:35:08.480476 25317 heartbeater.cc:503] Master 127.15.228.126:36589 was elected leader, sending a full tablet report...
W20260808 04:35:08.520128 25842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:08.520498 25842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:08.520547 25842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:08.524673 25842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:35:08.524732 25842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:08.524753 25842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:08.524770 25842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:35:08.524788 25842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:35:08.530673 25842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:36589,127.15.228.125:42221,127.15.228.124:36829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.124.keytab
--principal=oryx/127.15.228.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.124:36829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.228.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:08.531939 25842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:08.533020 25842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:08.545607 25962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:08.545789 25842 server_base.cc:1034] running on GCE node
W20260808 04:35:08.545611 25964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:08.545607 25961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:08.546752 25842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:08.547930 25842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:08.549124 25842 hybrid_clock.cc:648] HybridClock initialized: now 1786163708549096 us; error 41 us; skew 500 ppm
Aug 08 04:35:08 dist-test-slave-drl0 krb5kdc[24991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163708, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:08.552564 25842 init.cc:377] Logged in from keytab as oryx/127.15.228.124@KRBTEST.COM (short username oryx)
I20260808 04:35:08.553592 25842 webserver.cc:460] Webserver started at http://127.15.228.124:46505/ using document root <none> and password file <none>
I20260808 04:35:08.554193 25842 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:08.554287 25842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:08.554543 25842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:08.556877 25842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/data/instance:
uuid: "1b5893d92b424d85bc9b7a9581005d68"
format_stamp: "Formatted at 2026-08-08 04:35:08 on dist-test-slave-drl0"
server_key: "cbb757e44e17155ad31b22a4c27ee3dd"
server_key_iv: "2f7ced67ee49e44d6bf2803a1231019b"
server_key_version: "encryptionkey@0"
I20260808 04:35:08.557483 25842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/wal/instance:
uuid: "1b5893d92b424d85bc9b7a9581005d68"
format_stamp: "Formatted at 2026-08-08 04:35:08 on dist-test-slave-drl0"
server_key: "cbb757e44e17155ad31b22a4c27ee3dd"
server_key_iv: "2f7ced67ee49e44d6bf2803a1231019b"
server_key_version: "encryptionkey@0"
I20260808 04:35:08.561573 25842 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260808 04:35:08.564779 25975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:08.566660 25842 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260808 04:35:08.566833 25842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/wal
uuid: "1b5893d92b424d85bc9b7a9581005d68"
format_stamp: "Formatted at 2026-08-08 04:35:08 on dist-test-slave-drl0"
server_key: "cbb757e44e17155ad31b22a4c27ee3dd"
server_key_iv: "2f7ced67ee49e44d6bf2803a1231019b"
server_key_version: "encryptionkey@0"
I20260808 04:35:08.567010 25842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:08.610817 25842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:08.614580 25842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:08.614872 25842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:08.626250 25842 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.124:36829
I20260808 04:35:08.626649 26057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.124:36829 every 8 connection(s)
I20260808 04:35:08.627502 25842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/data/info.pb
I20260808 04:35:08.630716 26062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:08.635085 26062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:35:08.635488 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 25842
Aug 08 04:35:08 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163708, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.124@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM
Aug 08 04:35:08 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.228.124@KRBTEST.COM
I20260808 04:35:08.651016 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.637540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41502 (local address 127.15.228.126:36589)
0808 04:35:08.637740 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:08.637745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:08.637881 (+ 136us) server_negotiation.cc:407] Connection header received
0808 04:35:08.638971 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:08.638975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:08.639039 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:08.639140 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:08.640194 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.640733 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.641859 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.642075 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.642189 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.645621 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:08.645624 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:08.645627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:08.645659 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:08.648066 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.648621 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.648626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.648627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.648718 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.649211 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.649246 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.649250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.649542 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:08.649622 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:08.650682 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0808 04:35:08.650808 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260808 04:35:08.652195 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.637194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42730 (local address 127.15.228.124:36829)
0808 04:35:08.637900 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:08.637911 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:08.637950 (+ 39us) server_negotiation.cc:407] Connection header received
0808 04:35:08.638628 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:08.638645 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:08.639051 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:08.639260 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:08.640199 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.641163 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.642084 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.642477 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.642678 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.646483 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:08.646492 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:08.646508 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:08.646542 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:08.648720 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.649296 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.649304 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.649309 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.649402 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.649957 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.649960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.649961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.650255 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:08.650488 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:08.651124 (+ 636us) server_negotiation.cc:299] Negotiation successful
0808 04:35:08.651296 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"spinlock_wait_cycles":59136,"thread_start_us":159,"threads_started":1}
I20260808 04:35:08.652264 26064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.637218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.0.0.1:41502)
0808 04:35:08.637743 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:08.637784 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:08.638670 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:08.639195 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:08.639203 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:08.639661 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:08.640037 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.640052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.640930 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.640933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.641718 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.641728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.642232 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.642236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.642469 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.643165 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:08.643190 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:08.645465 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:08.648191 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.648200 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.648208 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.648507 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.648874 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.648877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.648879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.649070 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.649729 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:08.649741 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:08.650275 (+ 534us) client_negotiation.cc:769] Sending connection context
0808 04:35:08.650605 (+ 330us) client_negotiation.cc:240] Negotiation successful
0808 04:35:08.651215 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":164,"threads_started":1}
I20260808 04:35:08.653755 26062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:35:08 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163708, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.124@KRBTEST.COM for oryx/127.15.228.125@KRBTEST.COM
I20260808 04:35:08.664094 26064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.654349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:42221 (local address 127.0.0.1:34478)
0808 04:35:08.654499 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:08.654510 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:08.654571 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:08.655189 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:08.655191 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:08.655397 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:08.655519 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.655526 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.656526 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.656529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.657131 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.657140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.657774 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.657782 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.657863 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.658347 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:08.658359 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:08.659968 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:08.662119 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.662124 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.662126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.662450 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.662904 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.662907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.662909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.663113 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.663596 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:08.663599 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:08.663688 (+ 89us) client_negotiation.cc:769] Sending connection context
0808 04:35:08.663788 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 04:35:08.663915 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260808 04:35:08.664101 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.654477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34478 (local address 127.15.228.125:42221)
0808 04:35:08.654883 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:08.654888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:08.654902 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:35:08.654981 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:08.654984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:08.655037 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:08.655119 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:08.655702 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.656395 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.657334 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.657571 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.657664 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.660148 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:08.660151 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:08.660153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:08.660185 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:08.661914 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.662636 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.662639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.662640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.662723 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.663259 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.663263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.663264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.663462 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:08.663558 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:08.663848 (+ 290us) server_negotiation.cc:299] Negotiation successful
0808 04:35:08.663953 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":130,"threads_started":1}
I20260808 04:35:08.665434 26062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:35:08 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163708, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.124@KRBTEST.COM for oryx/127.15.228.124@KRBTEST.COM
I20260808 04:35:08.674731 26064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.666110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36829 (local address 127.0.0.1:42738)
0808 04:35:08.666242 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:08.666257 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:08.666352 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:08.666640 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:08.666643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:08.666798 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:08.666869 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.666884 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.667946 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.667948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.668592 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.668601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.669246 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.669248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.669379 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.669861 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:08.669873 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:08.671262 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:08.673271 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.673277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.673279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.673477 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.673747 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.673752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.673757 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.673833 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.674203 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:08.674207 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:08.674297 (+ 90us) client_negotiation.cc:769] Sending connection context
0808 04:35:08.674398 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 04:35:08.674537 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260808 04:35:08.674741 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.666290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42738 (local address 127.15.228.124:36829)
0808 04:35:08.666435 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:08.666439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:08.666451 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:35:08.666479 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:08.666480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:08.666513 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:08.666583 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:08.667158 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.667761 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.668803 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.669104 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.669198 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.671428 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:08.671431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:08.671433 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:08.671453 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:08.673146 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.673574 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.673579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.673581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.673655 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.673925 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.673930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.673931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.674101 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:08.674203 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:08.674457 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 04:35:08.674588 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":67}
I20260808 04:35:08.678552 26062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: Bootstrap starting.
I20260808 04:35:08.682549 26062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:08.684237 26062 log.cc:824] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:08.686808 26062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: No bootstrap required, opened a new log
I20260808 04:35:08.690451 26062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } }
I20260808 04:35:08.690658 26062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:08.690693 26062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b5893d92b424d85bc9b7a9581005d68, State: Initialized, Role: FOLLOWER
I20260808 04:35:08.691172 26062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } }
I20260808 04:35:08.691848 26070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } } }
I20260808 04:35:08.692010 26070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:35:08.692633 26062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 04:35:08.699119 26085 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:35:08.699213 26085 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:35:08.703179 25842 master_runner.cc:428] Detected that this master 1b5893d92b424d85bc9b7a9581005d68 is joining an existing cluster
I20260808 04:35:08.703248 25842 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.228.124:36829
I20260808 04:35:08.707850 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 25145
I20260808 04:35:08.716346 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.707001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41516 (local address 127.15.228.126:36589)
0808 04:35:08.707124 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:08.707129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:08.707143 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:35:08.707190 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:08.707194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:08.707241 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:08.707353 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:08.708071 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.708934 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.710583 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.710917 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.711044 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.712425 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:08.712430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:08.712434 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:08.712461 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:08.714382 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.714877 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.714883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.714885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.715007 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.715324 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.715328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.715329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.715553 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:08.715628 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:08.716039 (+ 411us) server_negotiation.cc:299] Negotiation successful
0808 04:35:08.716172 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:35:08.717139 26091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.706033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.0.0.1:41516)
0808 04:35:08.706748 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:08.706766 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:08.706867 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:08.707365 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:08.707370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:08.707638 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:08.707914 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.707923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.709083 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.709088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.710467 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.710474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.711211 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.711214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.711302 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.711889 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:08.711901 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:08.712270 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:08.714509 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.714512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.714515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.714762 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.715120 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.715122 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.715124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.715223 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.715707 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:08.715709 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:08.715799 (+ 90us) client_negotiation.cc:769] Sending connection context
0808 04:35:08.716613 (+ 814us) client_negotiation.cc:240] Negotiation successful
0808 04:35:08.716779 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":32,"thread_start_us":161,"threads_started":1}
I20260808 04:35:08.734326 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.706240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34488 (local address 127.15.228.125:42221)
0808 04:35:08.706392 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:08.706400 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:08.707144 (+ 744us) server_negotiation.cc:407] Connection header received
0808 04:35:08.707415 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:08.707419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:08.707476 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:08.707530 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:08.708252 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.708999 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.710386 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.710729 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.710799 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.729779 (+ 18980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:08.729787 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:08.729793 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:08.729835 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:08.731719 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.732638 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.732646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.732650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.732785 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.733118 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.733124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.733128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.733387 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:08.733458 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:08.733911 (+ 453us) server_negotiation.cc:299] Negotiation successful
0808 04:35:08.734080 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:35:08.735129 26093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.706040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:42221 (local address 127.0.0.1:34488)
0808 04:35:08.707059 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:08.707071 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:08.707289 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:08.707680 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:08.707682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:08.707962 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:08.708114 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.708121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.709127 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.709130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.710250 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.710262 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.711085 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.711089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.711200 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.728741 (+ 17541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:08.728771 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:08.729469 (+ 698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:08.731931 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.731937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.731942 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.732449 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.732904 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.732909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.732914 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.733022 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.733531 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:08.733536 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:08.733717 (+ 181us) client_negotiation.cc:769] Sending connection context
0808 04:35:08.734492 (+ 775us) client_negotiation.cc:240] Negotiation successful
0808 04:35:08.734861 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":952,"mutex_wait_us":90,"spinlock_wait_cycles":1408,"thread_start_us":145,"threads_started":1}
I20260808 04:35:08.740774 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.706711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42746 (local address 127.15.228.124:36829)
0808 04:35:08.706835 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:08.706839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:08.707173 (+ 334us) server_negotiation.cc:407] Connection header received
0808 04:35:08.711588 (+ 4415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:08.711592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:08.711642 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:08.711694 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:08.712919 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.713697 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.715074 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.715406 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.715475 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.722173 (+ 6698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:08.722181 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:08.722187 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:08.722224 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:08.737910 (+ 15686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.738507 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.738513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.738518 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.738595 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.739043 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.739050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.739052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.739424 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:08.739616 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:08.740287 (+ 671us) server_negotiation.cc:299] Negotiation successful
0808 04:35:08.740520 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:35:08.741632 25496 heartbeater.cc:503] Master 127.15.228.126:36589 was elected leader, sending a full tablet report...
I20260808 04:35:08.742069 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.706576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36829 (local address 127.0.0.1:42746)
0808 04:35:08.707102 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:08.707116 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:08.709426 (+ 2310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:08.712453 (+ 3027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:08.712455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:08.712646 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:08.712786 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.712795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.713844 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.713846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.714956 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.714967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.720705 (+ 5738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.720715 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.720799 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.721427 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:08.721448 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:08.721977 (+ 529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:08.738086 (+ 16109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.738092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.738098 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.738402 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.738693 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.738699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.738704 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.738786 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.739919 (+ 1133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:08.739925 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:08.740068 (+ 143us) client_negotiation.cc:769] Sending connection context
0808 04:35:08.741007 (+ 939us) client_negotiation.cc:240] Negotiation successful
0808 04:35:08.741182 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"spinlock_wait_cycles":4992,"thread_start_us":124,"threads_started":1}
I20260808 04:35:08.748713 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:45007
--local_ip_for_outbound_sockets=127.15.228.65
--tserver_master_addrs=127.15.228.126:36589
--webserver_port=39711
--webserver_interface=127.15.228.65
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.65.keytab
--principal=oryx/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.228.126:36589,127.15.228.125:42221
--tserver_master_addrs=127.15.228.126:36589,127.15.228.125:42221,127.15.228.124:36829 with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:35:08 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163707, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.126@KRBTEST.COM for oryx/127.15.228.124@KRBTEST.COM
I20260808 04:35:08.755887 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.745631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36829 (local address 127.0.0.1:42752)
0808 04:35:08.745771 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:08.745789 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:08.745876 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:08.746186 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:08.746191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:08.746426 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:08.746573 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.746584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.747647 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.747654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.748524 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.748535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.749359 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.749363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.749474 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.750304 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:08.750319 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:08.752116 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:08.754135 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.754139 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.754141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.754382 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.754677 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.754680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.754682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.754782 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.755347 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:08.755350 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:08.755446 (+ 96us) client_negotiation.cc:769] Sending connection context
0808 04:35:08.755556 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 04:35:08.755704 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 04:35:08.756522 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.745831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42752 (local address 127.15.228.124:36829)
0808 04:35:08.745943 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:08.745947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:08.745961 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:35:08.746028 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:08.746031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:08.746084 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:08.746217 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:08.746703 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.747522 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.748891 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.749231 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.749346 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.752298 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:08.752303 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:08.752306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:08.752348 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:08.754006 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.754492 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.754495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.754497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.754576 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.754885 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.754889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.754890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.755204 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:08.755300 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:08.756259 (+ 959us) server_negotiation.cc:299] Negotiation successful
0808 04:35:08.756377 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:35:08.758004 25052 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.228.124:36829: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68"
member_type: NON_VOTER
last_known_addr {
host: "127.15.228.124"
port: 36829
}
attrs {
promote: true
}
}
dest_uuid: "7eb94118f70c4552bcab3f523c207610"
cas_config_opid_index: 5
I20260808 04:35:08.758833 25052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: NON_VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: true } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: NON_VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: true } }
I20260808 04:35:08.760313 26095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1b5893d92b424d85bc9b7a9581005d68 (127.15.228.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: NON_VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: true } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: NON_VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: true } } }
I20260808 04:35:08.762480 26095 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.228.124:36829
I20260808 04:35:08.762696 25839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7eb94118f70c4552bcab3f523c207610" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: NON_VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: true } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: NON_VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: true } } }
I20260808 04:35:08.762812 25839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:08.763809 25842 master.cc:537] Master@127.15.228.124:36829 shutting down...
I20260808 04:35:08.764042 26019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:08.765677 26019 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 1 FOLLOWER]: Refusing update from remote peer 7eb94118f70c4552bcab3f523c207610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 04:35:08.766181 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.760593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:42221 (local address 127.0.0.1:34504)
0808 04:35:08.760913 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:08.760930 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:08.761041 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:08.761869 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:08.761874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:08.761896 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:08.762012 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.762022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.763222 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.763226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.764405 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.764416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.765731 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.765738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.765828 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.765874 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 04:35:08.765952 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 04:35:08.766029 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260808 04:35:08.766722 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.761234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.15.228.125:42221)
0808 04:35:08.761394 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:08.761400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:08.761419 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:35:08.761483 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:08.761489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:08.761575 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:08.761685 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:08.762168 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.763097 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.764557 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.765263 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.765508 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.765653 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:08.766464 (+ 811us) server_negotiation.cc:299] Negotiation successful
0808 04:35:08.766567 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:35:08.767230 25906 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 1 LEARNER]: Refusing update from remote peer 7eb94118f70c4552bcab3f523c207610: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 04:35:08.767844 26095 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: NON_VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 04:35:08.769027 25839 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: NON_VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:35:08.770138 25906 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1b5893d92b424d85bc9b7a9581005d68 (127.15.228.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: NON_VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: true } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: NON_VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: true } } }
W20260808 04:35:08.770875 25032 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 -> Peer 1b5893d92b424d85bc9b7a9581005d68 (127.15.228.124:36829): Couldn't send request to peer 1b5893d92b424d85bc9b7a9581005d68. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 04:35:08.772615 25842 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:35:08.772907 25842 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:08.773032 25842 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: stopping tablet replica
I20260808 04:35:08.774122 25944 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eb94118f70c4552bcab3f523c207610. Latest consensus state: current_term: 1 leader_uuid: "7eb94118f70c4552bcab3f523c207610" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: NON_VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: true } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: NON_VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: true } } }
I20260808 04:35:08.774253 25944 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [sys.catalog]: This master's current role is: LEARNER
I20260808 04:35:08.774932 25839 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610: attempting to promote NON_VOTER fd4721373d7b4a7686b043b6116850eb to VOTER
I20260808 04:35:08.775442 26097 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7eb94118f70c4552bcab3f523c207610" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: NON_VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: true } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: NON_VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: true } } }
I20260808 04:35:08.775475 25839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: NON_VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: true } }
I20260808 04:35:08.775559 26097 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [sys.catalog]: This master's current role is: LEARNER
I20260808 04:35:08.777009 25905 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 1 LEARNER]: Refusing update from remote peer 7eb94118f70c4552bcab3f523c207610: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260808 04:35:08.777323 25032 connection.cc:535] client connection to 127.15.228.124:36829 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 04:35:08.777412 25032 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 -> Peer 1b5893d92b424d85bc9b7a9581005d68 (127.15.228.124:36829): Couldn't send request to peer 1b5893d92b424d85bc9b7a9581005d68. Status: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20260808 04:35:08.777474 25838 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 04:35:08.779608 26095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fd4721373d7b4a7686b043b6116850eb (127.15.228.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: NON_VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: true } } }
I20260808 04:35:08.780172 25905 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fd4721373d7b4a7686b043b6116850eb (127.15.228.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: NON_VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: true } } }
I20260808 04:35:08.783691 26095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7eb94118f70c4552bcab3f523c207610" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: NON_VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: true } } }
I20260808 04:35:08.783825 25838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7eb94118f70c4552bcab3f523c207610" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: NON_VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: true } } }
I20260808 04:35:08.783923 25838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:08.783918 26097 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7eb94118f70c4552bcab3f523c207610" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: NON_VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: true } } }
I20260808 04:35:08.784098 26097 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:35:08.783833 26095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:08.790728 25842 master.cc:552] Master@127.15.228.124:36829 shutdown complete.
I20260808 04:35:08.790808 25842 master_runner.cc:305] Clearing existing system tablet
I20260808 04:35:08.791139 25842 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:08.791213 25842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:08.792332 25842 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 04:35:08.793299 26103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:08.793629 25842 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:08.793819 25842 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 04:35:08.793871 25842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/wal
uuid: "1b5893d92b424d85bc9b7a9581005d68"
format_stamp: "Formatted at 2026-08-08 04:35:08 on dist-test-slave-drl0"
server_key: "cbb757e44e17155ad31b22a4c27ee3dd"
server_key_iv: "2f7ced67ee49e44d6bf2803a1231019b"
server_key_version: "encryptionkey@0"
I20260808 04:35:08.796187 25842 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:08.798128 25842 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 04:35:08.798239 25842 log.cc:1196] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 04:35:08.798499 25842 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: Deleting consensus metadata
I20260808 04:35:08.798686 25842 master_runner.cc:315] Copying system tablet from 127.15.228.126:36589
I20260808 04:35:08.799988 25842 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: tablet copy: Beginning tablet copy session from remote peer at address 127.15.228.126:36589
I20260808 04:35:08.808683 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.800629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41518 (local address 127.15.228.126:36589)
0808 04:35:08.800765 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:08.800771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:08.800976 (+ 205us) server_negotiation.cc:407] Connection header received
0808 04:35:08.801160 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:08.801163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:08.801219 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:08.801312 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:08.801780 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.802327 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.803458 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.803721 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.803838 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.805125 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:08.805128 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:08.805131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:08.805155 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:08.806737 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.807235 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.807239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.807241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.807330 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:08.807628 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:08.807632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:08.807634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:08.807962 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:08.808098 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:08.808403 (+ 305us) server_negotiation.cc:299] Negotiation successful
0808 04:35:08.808517 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:35:08.808701 26114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:08.800551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.0.0.1:41518)
0808 04:35:08.800867 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:08.800880 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:08.801047 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:08.801330 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:08.801334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:08.801553 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:08.801656 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.801662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.802450 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.802454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.803314 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:08.803325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:08.803847 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:08.803851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:08.803949 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:08.804603 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:08.804620 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:08.804980 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:08.806858 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.806863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.806865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.807118 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.807439 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:08.807442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:08.807444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:08.807532 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:08.808097 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:08.808100 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:08.808211 (+ 111us) client_negotiation.cc:769] Sending connection context
0808 04:35:08.808322 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 04:35:08.808477 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":89,"threads_started":1}
I20260808 04:35:08.809466 25082 tablet_copy_service.cc:140] P 7eb94118f70c4552bcab3f523c207610: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1b5893d92b424d85bc9b7a9581005d68 ({username='oryx', principal='oryx/127.15.228.124@KRBTEST.COM'} at 127.0.0.1:41518)
I20260808 04:35:08.809587 25082 tablet_copy_service.cc:161] P 7eb94118f70c4552bcab3f523c207610: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1b5893d92b424d85bc9b7a9581005d68 at {username='oryx', principal='oryx/127.15.228.124@KRBTEST.COM'} at 127.0.0.1:41518: session id = 1b5893d92b424d85bc9b7a9581005d68-00000000000000000000000000000000
I20260808 04:35:08.810447 25082 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 04:35:08.811677 25842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 04:35:08.813661 25842 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: tablet copy: Starting download of 0 data blocks...
I20260808 04:35:08.813961 25842 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: tablet copy: Starting download of 1 WAL segments...
I20260808 04:35:08.815577 25842 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 04:35:08.817456 25082 tablet_copy_service.cc:342] P 7eb94118f70c4552bcab3f523c207610: Request end of tablet copy session 1b5893d92b424d85bc9b7a9581005d68-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.228.124@KRBTEST.COM'} at 127.0.0.1:41518
I20260808 04:35:08.817559 25082 tablet_copy_service.cc:433] P 7eb94118f70c4552bcab3f523c207610: ending tablet copy session 1b5893d92b424d85bc9b7a9581005d68-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1b5893d92b424d85bc9b7a9581005d68
I20260808 04:35:08.825192 25842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:08.827126 26115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:08.827142 26116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:08.835115 26119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:08.835359 25842 server_base.cc:1034] running on GCE node
I20260808 04:35:08.835647 25842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:08.836015 25842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:08.837227 25842 hybrid_clock.cc:648] HybridClock initialized: now 1786163708837220 us; error 38 us; skew 500 ppm
Aug 08 04:35:08 dist-test-slave-drl0 krb5kdc[24991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163708, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:08.839231 25842 init.cc:377] Logged in from keytab as oryx/127.15.228.124@KRBTEST.COM (short username oryx)
I20260808 04:35:08.839754 25842 webserver.cc:460] Webserver started at http://127.15.228.124:42095/ using document root <none> and password file <none>
I20260808 04:35:08.839969 25842 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:08.840025 25842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:08.841189 25842 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 04:35:08.842190 26127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:08.842569 25842 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 04:35:08.842661 25842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/wal
uuid: "1b5893d92b424d85bc9b7a9581005d68"
format_stamp: "Formatted at 2026-08-08 04:35:08 on dist-test-slave-drl0"
server_key: "cbb757e44e17155ad31b22a4c27ee3dd"
server_key_iv: "2f7ced67ee49e44d6bf2803a1231019b"
server_key_version: "encryptionkey@0"
I20260808 04:35:08.842751 25842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:08.863545 25842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:08.866353 25842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:08.872567 25842 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.124:36829
I20260808 04:35:08.873178 25842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/master-2/data/info.pb
I20260808 04:35:08.874830 26196 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:35:08.875797 26196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: Bootstrap starting.
I20260808 04:35:08.879074 26195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.124:36829 every 8 connection(s)
I20260808 04:35:08.887259 26196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:35:08.887593 26196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: Bootstrap complete.
I20260808 04:35:08.888546 26196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: NON_VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: true } }
I20260808 04:35:08.888700 26196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b5893d92b424d85bc9b7a9581005d68, State: Initialized, Role: LEARNER
I20260808 04:35:08.888823 26196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: NON_VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: true } }
I20260808 04:35:08.889321 26201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: NON_VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: true } } }
I20260808 04:35:08.889503 26196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:35:08.889524 26201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [sys.catalog]: This master's current role is: LEARNER
I20260808 04:35:08.892681 26218 catalog_manager.cc:1259] Loaded cluster ID: 7b0fb3629a754d6186ff6c61f948734c
I20260808 04:35:08.892822 26218 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: loading cluster ID for follower catalog manager: success
I20260808 04:35:08.894511 26218 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: acquiring CA information for follower catalog manager: success
I20260808 04:35:08.895695 26218 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 04:35:08.923854 26094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:08.924265 26094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:08.924403 26094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:08.932400 26094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:08.932538 26094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:08.932696 26094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:35:08.941012 26094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.65.keytab
--principal=oryx/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:45007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=39711
--tserver_master_addrs=127.15.228.126:36589,127.15.228.125:42221,127.15.228.124:36829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:08.942477 26094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:08.944597 26094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:08.957238 26227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:08.957247 26225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:08.957324 26224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:08.958021 26094 server_base.cc:1034] running on GCE node
I20260808 04:35:08.958663 26094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:08.960122 26094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:08.961385 26094 hybrid_clock.cc:648] HybridClock initialized: now 1786163708961345 us; error 72 us; skew 500 ppm
Aug 08 04:35:08 dist-test-slave-drl0 krb5kdc[24991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163708, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:08.964830 26094 init.cc:377] Logged in from keytab as oryx/127.15.228.65@KRBTEST.COM (short username oryx)
I20260808 04:35:08.966190 26094 webserver.cc:460] Webserver started at http://127.15.228.65:39711/ using document root <none> and password file <none>
I20260808 04:35:08.967398 26094 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:08.967525 26094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:08.972764 26094 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:35:08.975382 26236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:08.977051 26094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:35:08.977207 26094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "0eba8ed56d6047daa77802d8f0ffc71c"
format_stamp: "Formatted at 2026-08-08 04:35:07 on dist-test-slave-drl0"
server_key: "6444f345b9f338e3a580389faa935cf2"
server_key_iv: "d4f4106dc268c3722f92c0130f95f0f6"
server_key_version: "encryptionkey@0"
I20260808 04:35:08.977783 26094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:08.992574 25692 heartbeater.cc:503] Master 127.15.228.126:36589 was elected leader, sending a full tablet report...
I20260808 04:35:09.010324 26094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:09.013772 26094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:09.014027 26094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:09.014796 26094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:09.015981 26094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:09.016038 26094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:09.016081 26094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:09.016137 26094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:09.032474 26094 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:45007
I20260808 04:35:09.032557 26383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:45007 every 8 connection(s)
I20260808 04:35:09.033571 26094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:35:09.040866 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 26094
I20260808 04:35:09.041087 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 25323
Aug 08 04:35:09 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163708, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM
Aug 08 04:35:09 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163708, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.125@KRBTEST.COM
Aug 08 04:35:09 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163708, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.124@KRBTEST.COM
I20260808 04:35:09.055598 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.037729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:57031 (local address 127.15.228.126:36589)
0808 04:35:09.037877 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.037881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.037968 (+ 87us) server_negotiation.cc:407] Connection header received
0808 04:35:09.038611 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.038617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.038674 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.038725 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:09.040235 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.041348 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.043475 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.043831 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.043958 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.049495 (+ 5537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:09.049499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:09.049502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:09.049534 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:09.051930 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.053403 (+ 1473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.053409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.053412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.053521 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.054040 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.054044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.054046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.054332 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:09.054427 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.055269 (+ 842us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.055421 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:35:09.055757 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.035700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:55893 (local address 127.15.228.125:42221)
0808 04:35:09.035856 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.035861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.037105 (+ 1244us) server_negotiation.cc:407] Connection header received
0808 04:35:09.038497 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.038501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.038573 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.038653 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:09.040311 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.041348 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.044663 (+ 3315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.045178 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.045293 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.049491 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:09.049495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:09.049498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:09.049533 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:09.052827 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.053561 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.053568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.053573 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.053688 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.054106 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.054112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.054117 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.054446 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:09.054555 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.055322 (+ 767us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.055498 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 04:35:09.056666 26392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.035612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:48035 (local address 127.15.228.124:36829)
0808 04:35:09.035899 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.035986 (+ 87us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.037104 (+ 1118us) server_negotiation.cc:407] Connection header received
0808 04:35:09.038605 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.038610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.038674 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.038764 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:09.040298 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.041401 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.044173 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.044737 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.045022 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.050248 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:09.050256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:09.050262 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:09.050304 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:09.054376 (+ 4072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.054842 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.054846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.054848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.054968 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.055277 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.055281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.055282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.055520 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:09.055616 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.056387 (+ 771us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.056513 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":88,"threads_started":1}
I20260808 04:35:09.057441 26396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.037458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.65:57031)
0808 04:35:09.037884 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:09.037897 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:09.038264 (+ 367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:09.038895 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:09.038898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:09.039442 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:09.040077 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.040092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.041789 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.041794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.043288 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.043307 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.044283 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.044288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.044618 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.045537 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:09.045559 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:09.049237 (+ 3678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:09.052710 (+ 3473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:09.052720 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:09.052744 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:09.053219 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:09.053680 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:09.053683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:09.053685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:09.053915 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:09.054503 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:09.054510 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:09.054950 (+ 440us) client_negotiation.cc:769] Sending connection context
0808 04:35:09.055909 (+ 959us) client_negotiation.cc:240] Negotiation successful
0808 04:35:09.056391 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":123,"spinlock_wait_cycles":2944,"thread_start_us":188,"threads_started":1}
I20260808 04:35:09.057453 26393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.036171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:42221 (local address 127.15.228.65:55893)
0808 04:35:09.036925 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:09.036989 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:09.038237 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:09.038888 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:09.038898 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:09.039486 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:09.040160 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.040166 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.043529 (+ 3363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.043532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.044491 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.044503 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.045504 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.045507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.045788 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.046663 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:09.046699 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:09.049237 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:09.052988 (+ 3751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:09.052996 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:09.053001 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:09.053396 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:09.053832 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:09.053838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:09.053843 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:09.053967 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:09.054634 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:09.054640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:09.054973 (+ 333us) client_negotiation.cc:769] Sending connection context
0808 04:35:09.056019 (+ 1046us) client_negotiation.cc:240] Negotiation successful
0808 04:35:09.056916 (+ 897us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":849,"spinlock_wait_cycles":63616,"thread_start_us":251,"threads_started":1}
I20260808 04:35:09.058123 26394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.036170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36829 (local address 127.15.228.65:48035)
0808 04:35:09.036937 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:09.036989 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:09.038237 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:09.038962 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:09.038965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:09.039486 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:09.040142 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.040154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.041667 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.041671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.043977 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.043998 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.045500 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.045508 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.045781 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.046775 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:09.046800 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:09.050012 (+ 3212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:09.054494 (+ 4482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:09.054496 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:09.054498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:09.054725 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:09.055077 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:09.055080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:09.055082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:09.055174 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:09.055780 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:09.055786 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:09.055977 (+ 191us) client_negotiation.cc:769] Sending connection context
0808 04:35:09.056070 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 04:35:09.056522 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":342,"spinlock_wait_cycles":51584,"thread_start_us":246,"threads_started":1}
I20260808 04:35:09.058770 26386 heartbeater.cc:347] Connected to a master server at 127.15.228.125:42221
I20260808 04:35:09.059079 26385 heartbeater.cc:347] Connected to a master server at 127.15.228.126:36589
I20260808 04:35:09.059096 26386 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:09.059221 26385 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:09.059877 26386 heartbeater.cc:511] Master 127.15.228.125:42221 requested a full tablet report, sending...
I20260808 04:35:09.059877 26385 heartbeater.cc:511] Master 127.15.228.126:36589 requested a full tablet report, sending...
I20260808 04:35:09.061503 25052 ts_manager.cc:194] Re-registered known tserver with Master: 0eba8ed56d6047daa77802d8f0ffc71c (127.15.228.65:45007)
I20260808 04:35:09.062485 25052 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:57031
I20260808 04:35:09.072829 26384 heartbeater.cc:347] Connected to a master server at 127.15.228.124:36829
I20260808 04:35:09.072808 25893 ts_manager.cc:194] Registered new tserver with Master: 0eba8ed56d6047daa77802d8f0ffc71c (127.15.228.65:45007)
I20260808 04:35:09.072940 26384 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:09.073189 26384 heartbeater.cc:511] Master 127.15.228.124:36829 requested a full tablet report, sending...
I20260808 04:35:09.074396 26147 ts_manager.cc:194] Registered new tserver with Master: 0eba8ed56d6047daa77802d8f0ffc71c (127.15.228.65:45007)
I20260808 04:35:09.078853 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:34177
--local_ip_for_outbound_sockets=127.15.228.66
--tserver_master_addrs=127.15.228.126:36589
--webserver_port=36231
--webserver_interface=127.15.228.66
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.66.keytab
--principal=oryx/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.228.126:36589,127.15.228.125:42221
--tserver_master_addrs=127.15.228.126:36589,127.15.228.125:42221,127.15.228.124:36829 with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:09.213289 26397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:09.213575 26397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:09.213620 26397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:09.217921 26397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:09.218021 26397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:09.218114 26397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:35:09.224337 26397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.66.keytab
--principal=oryx/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:34177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=36231
--tserver_master_addrs=127.15.228.126:36589,127.15.228.125:42221,127.15.228.124:36829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:09.225683 26397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:09.227502 26397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:09.238588 26407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:09.238595 26409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:09.238575 26405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:09.239432 26397 server_base.cc:1034] running on GCE node
I20260808 04:35:09.239872 26397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:09.240527 26397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:09.241878 26397 hybrid_clock.cc:648] HybridClock initialized: now 1786163709241856 us; error 40 us; skew 500 ppm
I20260808 04:35:09.244607 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.238986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36829 (local address 127.0.0.1:42756)
0808 04:35:09.239124 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:09.239168 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:09.239288 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:09.240414 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:09.240419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:09.240450 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:09.240603 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.240610 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.241543 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.241548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.242821 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.242836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.244157 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.244160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.244231 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.244252 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 04:35:09.244366 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 04:35:09.244450 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 04:35:09.244652 26392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.239510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42756 (local address 127.15.228.124:36829)
0808 04:35:09.239651 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.239810 (+ 159us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.239864 (+ 54us) server_negotiation.cc:407] Connection header received
0808 04:35:09.239966 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.239969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.240203 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.240381 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:09.240771 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.241417 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.243055 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.243767 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.243959 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.244095 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.244384 (+ 289us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.244467 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:35:09.245775 26159 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
Aug 08 04:35:09 dist-test-slave-drl0 krb5kdc[24991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163709, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:09.246239 26201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eb94118f70c4552bcab3f523c207610. Latest consensus state: current_term: 1 leader_uuid: "7eb94118f70c4552bcab3f523c207610" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: NON_VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: true } } }
I20260808 04:35:09.246431 26201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [sys.catalog]: This master's current role is: LEARNER
I20260808 04:35:09.246687 26397 init.cc:377] Logged in from keytab as oryx/127.15.228.66@KRBTEST.COM (short username oryx)
I20260808 04:35:09.247705 26397 webserver.cc:460] Webserver started at http://127.15.228.66:36231/ using document root <none> and password file <none>
I20260808 04:35:09.248315 26397 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:09.248412 26397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:09.252589 26397 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260808 04:35:09.254791 26419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:09.256474 26397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:35:09.256651 26397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "93fc6b49759b480a8cd03c65ecb21bb9"
format_stamp: "Formatted at 2026-08-08 04:35:07 on dist-test-slave-drl0"
server_key: "1c45af87e145d521a15c4ba19858405f"
server_key_iv: "104389daac7ab9bce2ed76f530e5a772"
server_key_version: "encryptionkey@0"
I20260808 04:35:09.257239 26397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:09.294727 26397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:09.299291 26397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:09.299546 26397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:09.300350 26397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:09.301398 26397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:09.301461 26397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:09.301509 26397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:09.301528 26397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:09.312811 26397 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:34177
I20260808 04:35:09.312932 26573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:34177 every 8 connection(s)
I20260808 04:35:09.313889 26397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:35:09.319722 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 26397
I20260808 04:35:09.319952 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 25506
Aug 08 04:35:09 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163709, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.124@KRBTEST.COM
I20260808 04:35:09.339291 26392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.316068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:40621 (local address 127.15.228.124:36829)
0808 04:35:09.316215 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.316219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.317256 (+ 1037us) server_negotiation.cc:407] Connection header received
0808 04:35:09.318782 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.318786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.318858 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.319099 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:09.326805 (+ 7706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.328120 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.330888 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.331269 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.331370 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.335010 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:09.335016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:09.335022 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:09.335057 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:09.336722 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.337368 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.337374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.337379 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.337462 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.337845 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.337851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.337855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.338103 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:09.338172 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.338884 (+ 712us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.339048 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:35:09.341135 26579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.316481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36829 (local address 127.15.228.66:40621)
0808 04:35:09.317052 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:09.317110 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:09.318485 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:09.319340 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:09.319351 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:09.323363 (+ 4012us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:09.326561 (+ 3198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.326580 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.328578 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.328586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.330683 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.330705 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.331627 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.331633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.331980 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.332782 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:09.332807 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:09.334748 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:09.336874 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:09.336882 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:09.336903 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:09.337247 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:09.337565 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:09.337570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:09.337574 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:09.337748 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:09.338248 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:09.338257 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:09.338611 (+ 354us) client_negotiation.cc:769] Sending connection context
0808 04:35:09.339566 (+ 955us) client_negotiation.cc:240] Negotiation successful
0808 04:35:09.339944 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":712,"threads_started":1}
I20260808 04:35:09.343415 26575 heartbeater.cc:347] Connected to a master server at 127.15.228.124:36829
I20260808 04:35:09.343880 26575 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:09.344883 26575 heartbeater.cc:511] Master 127.15.228.124:36829 requested a full tablet report, sending...
I20260808 04:35:09.346735 26147 ts_manager.cc:194] Registered new tserver with Master: 93fc6b49759b480a8cd03c65ecb21bb9 (127.15.228.66:34177)
Aug 08 04:35:09 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163709, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM
Aug 08 04:35:09 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163709, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.125@KRBTEST.COM
I20260808 04:35:09.351389 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:38279
--local_ip_for_outbound_sockets=127.15.228.67
--tserver_master_addrs=127.15.228.126:36589
--webserver_port=35127
--webserver_interface=127.15.228.67
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.67.keytab
--principal=oryx/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.228.126:36589,127.15.228.125:42221
--tserver_master_addrs=127.15.228.126:36589,127.15.228.125:42221,127.15.228.124:36829 with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:35:09.352593 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.317759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:54435 (local address 127.15.228.126:36589)
0808 04:35:09.317927 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.317933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.318068 (+ 135us) server_negotiation.cc:407] Connection header received
0808 04:35:09.319144 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.319149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.319205 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.319331 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:09.340714 (+ 21383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.341538 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.343228 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.343682 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.343814 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.347726 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:09.347734 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:09.347740 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:09.347788 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:09.350319 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.351131 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.351135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.351139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.351231 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.351604 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.351610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.351614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.351875 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:09.352001 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.352283 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.352407 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 04:35:09.352613 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.317636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.66:54435)
0808 04:35:09.317968 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:09.317979 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:09.318505 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:09.319355 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:09.319360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:09.324912 (+ 5552us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:09.325864 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.325950 (+ 86us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.341690 (+ 15740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.341696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.342996 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.343018 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.344083 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.344099 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.344280 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.345191 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:09.345214 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:09.347472 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:09.350485 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:09.350492 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:09.350497 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:09.350994 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:09.351337 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:09.351340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:09.351342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:09.351442 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:09.351989 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:09.351992 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:09.352103 (+ 111us) client_negotiation.cc:769] Sending connection context
0808 04:35:09.352218 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 04:35:09.352411 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":91,"threads_started":1}
I20260808 04:35:09.352941 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.316103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:39099 (local address 127.15.228.125:42221)
0808 04:35:09.316265 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.316270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.317669 (+ 1399us) server_negotiation.cc:407] Connection header received
0808 04:35:09.319066 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.319078 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.319140 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.319224 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:09.342068 (+ 22844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.343026 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.344936 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.345289 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.345409 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.348865 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:09.348875 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:09.348877 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:09.348906 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:09.350509 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.351089 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.351094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.351097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.351219 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.351782 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.351788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.351792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.352068 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:09.352160 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.352538 (+ 378us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.352682 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 04:35:09.353530 26580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.316481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:42221 (local address 127.15.228.66:39099)
0808 04:35:09.317562 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:09.317578 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:09.318485 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:09.340407 (+ 21922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:09.340414 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:09.340729 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:09.341157 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.341197 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.343500 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.343505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.344727 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.344739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.345626 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.345630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.345795 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.346446 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:09.346461 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:09.348541 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:09.350651 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:09.350655 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:09.350657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:09.350992 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:09.351553 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:09.351558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:09.351563 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:09.351655 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:09.352227 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:09.352231 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:09.352341 (+ 110us) client_negotiation.cc:769] Sending connection context
0808 04:35:09.353102 (+ 761us) client_negotiation.cc:240] Negotiation successful
0808 04:35:09.353287 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"mutex_wait_us":26,"thread_start_us":161,"threads_started":1}
I20260808 04:35:09.353860 26576 heartbeater.cc:347] Connected to a master server at 127.15.228.126:36589
I20260808 04:35:09.353971 26576 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:09.354141 26577 heartbeater.cc:347] Connected to a master server at 127.15.228.125:42221
I20260808 04:35:09.354207 26576 heartbeater.cc:511] Master 127.15.228.126:36589 requested a full tablet report, sending...
I20260808 04:35:09.354239 26577 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:09.354418 26577 heartbeater.cc:511] Master 127.15.228.125:42221 requested a full tablet report, sending...
I20260808 04:35:09.354998 25052 ts_manager.cc:194] Re-registered known tserver with Master: 93fc6b49759b480a8cd03c65ecb21bb9 (127.15.228.66:34177)
I20260808 04:35:09.355244 25893 ts_manager.cc:194] Registered new tserver with Master: 93fc6b49759b480a8cd03c65ecb21bb9 (127.15.228.66:34177)
I20260808 04:35:09.355734 25052 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:54435
W20260808 04:35:09.491339 26583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:09.491681 26583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:09.491770 26583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:09.495844 26583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:09.495923 26583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:09.496021 26583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:35:09.502110 26583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.67.keytab
--principal=oryx/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:38279
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=35127
--tserver_master_addrs=127.15.228.126:36589,127.15.228.125:42221,127.15.228.124:36829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:09.503858 26583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:09.505095 26583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:09.515497 26594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:09.515497 26591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:09.515614 26592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:09.516062 26583 server_base.cc:1034] running on GCE node
I20260808 04:35:09.516489 26583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:09.517204 26583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:09.518384 26583 hybrid_clock.cc:648] HybridClock initialized: now 1786163709518355 us; error 43 us; skew 500 ppm
Aug 08 04:35:09 dist-test-slave-drl0 krb5kdc[24991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163709, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:09.521737 26583 init.cc:377] Logged in from keytab as oryx/127.15.228.67@KRBTEST.COM (short username oryx)
I20260808 04:35:09.522703 26583 webserver.cc:460] Webserver started at http://127.15.228.67:35127/ using document root <none> and password file <none>
I20260808 04:35:09.523355 26583 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:09.523463 26583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:09.527885 26583 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:35:09.531013 26603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:09.532361 26583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:35:09.532575 26583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "2114de6f886e4314b4d47f30c9cea294"
format_stamp: "Formatted at 2026-08-08 04:35:07 on dist-test-slave-drl0"
server_key: "3e7193f53e777b84c16a81447b3c88dd"
server_key_iv: "8333bccb76b4642955be28793fc423fb"
server_key_version: "encryptionkey@0"
I20260808 04:35:09.533550 26583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:09.557691 26583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:09.562819 26583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:09.563256 26583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:09.564185 26583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:09.565853 26583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:09.565922 26583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:09.565970 26583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:09.566051 26583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:09.577780 26583 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:38279
I20260808 04:35:09.577870 26758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:38279 every 8 connection(s)
I20260808 04:35:09.579085 26583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:35:09.583143 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 26583
Aug 08 04:35:09 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163709, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.124@KRBTEST.COM
Aug 08 04:35:09 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163709, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM
Aug 08 04:35:09 dist-test-slave-drl0 krb5kdc[24991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163709, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.125@KRBTEST.COM
I20260808 04:35:09.600307 26392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.582687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:55027 (local address 127.15.228.124:36829)
0808 04:35:09.582812 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.582818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.582945 (+ 127us) server_negotiation.cc:407] Connection header received
0808 04:35:09.584108 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.584112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.584186 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.584293 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:09.585875 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.586675 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.588675 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.589044 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.589134 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.593015 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:09.593021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:09.593026 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:09.593057 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:09.596556 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.597956 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.597960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.597962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.598059 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.598524 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.598530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.598532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.598995 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:09.599087 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.599837 (+ 750us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.599970 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:35:09.601109 26779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.586870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42766 (local address 127.15.228.124:36829)
0808 04:35:09.587259 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.587264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.587278 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:35:09.587316 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.587318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.587361 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.588428 (+ 1067us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:09.588534 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.589316 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.590679 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.591309 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.591462 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.594310 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:09.594337 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:09.594344 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:09.594413 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:09.598675 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.599378 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.599382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.599383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.599472 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.599789 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.599793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.599794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.599984 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:09.600233 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.600821 (+ 588us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.600941 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":94,"threads_started":1}
I20260808 04:35:09.601836 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.581118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:54701 (local address 127.15.228.126:36589)
0808 04:35:09.581287 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.581293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.582284 (+ 991us) server_negotiation.cc:407] Connection header received
0808 04:35:09.584038 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.584042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.584097 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.584152 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:09.587407 (+ 3255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.588214 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.589326 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.589643 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.589719 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.597227 (+ 7508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:09.597231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:09.597233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:09.597263 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:09.599634 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.600157 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.600162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.600165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.600261 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.600629 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.600634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.600636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.600994 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:09.601086 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.601508 (+ 422us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.601662 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 04:35:09.602602 26767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.582556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36829 (local address 127.15.228.67:55027)
0808 04:35:09.582843 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:09.582857 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:09.583532 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:09.584463 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:09.584466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:09.584917 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:09.585693 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.585712 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.586820 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.586823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.588543 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.588552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.589321 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.589325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.589601 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.590356 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:09.590381 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:09.592805 (+ 2424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:09.596699 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:09.596707 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:09.596725 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:09.597210 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:09.598186 (+ 976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:09.598189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:09.598194 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:09.598400 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:09.599164 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:09.599171 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:09.599539 (+ 368us) client_negotiation.cc:769] Sending connection context
0808 04:35:09.600750 (+ 1211us) client_negotiation.cc:240] Negotiation successful
0808 04:35:09.601120 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":172,"thread_start_us":97,"threads_started":1}
I20260808 04:35:09.603096 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.581113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:32847 (local address 127.15.228.125:42221)
0808 04:35:09.581283 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.581288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.582545 (+ 1257us) server_negotiation.cc:407] Connection header received
0808 04:35:09.583775 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.583780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.583848 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.583949 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:09.585888 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.586677 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.588680 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.589050 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.589127 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.594854 (+ 5727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:09.594857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:09.594860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:09.594886 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:09.596804 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.601028 (+ 4224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.601035 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.601040 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.601141 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.602066 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.602070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.602072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.602288 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:09.602378 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.602771 (+ 393us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.602898 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 04:35:09.603183 26764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.581566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:42221 (local address 127.15.228.67:32847)
0808 04:35:09.582430 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:09.582443 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:09.583532 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:09.584166 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:09.584174 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:09.585101 (+ 927us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:09.585689 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.585713 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.586843 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.586846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.588546 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.588557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.589321 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.589325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.589612 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.590811 (+ 1199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:09.590826 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:09.594669 (+ 3843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:09.596912 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:09.596915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:09.596917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:09.600764 (+ 3847us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:09.601248 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:09.601252 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:09.601254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:09.601342 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:09.602439 (+ 1097us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:09.602442 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:09.602530 (+ 88us) client_negotiation.cc:769] Sending connection context
0808 04:35:09.602632 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 04:35:09.602957 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":287,"spinlock_wait_cycles":8320,"thread_start_us":145,"threads_started":1}
I20260808 04:35:09.602609 26765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.581566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.67:54701)
0808 04:35:09.582101 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:09.582155 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:09.583862 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:09.584315 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:09.584319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:09.584927 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:09.587249 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.587259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.588366 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.588371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.589192 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.589204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.589915 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.589922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.590198 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.590602 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:09.590619 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:09.596017 (+ 5398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:09.599745 (+ 3728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:09.599751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:09.599755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:09.600027 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:09.600407 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:09.600410 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:09.600412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:09.600508 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:09.601152 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:09.601155 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:09.601292 (+ 137us) client_negotiation.cc:769] Sending connection context
0808 04:35:09.601994 (+ 702us) client_negotiation.cc:240] Negotiation successful
0808 04:35:09.602161 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":19,"spinlock_wait_cycles":4224,"thread_start_us":151,"threads_started":1}
I20260808 04:35:09.604635 26762 heartbeater.cc:347] Connected to a master server at 127.15.228.125:42221
I20260808 04:35:09.604754 26759 heartbeater.cc:347] Connected to a master server at 127.15.228.124:36829
I20260808 04:35:09.604889 26762 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:09.604888 26759 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:09.604921 26761 heartbeater.cc:347] Connected to a master server at 127.15.228.126:36589
I20260808 04:35:09.605034 26761 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:09.605593 26759 heartbeater.cc:511] Master 127.15.228.124:36829 requested a full tablet report, sending...
I20260808 04:35:09.605613 26762 heartbeater.cc:511] Master 127.15.228.125:42221 requested a full tablet report, sending...
I20260808 04:35:09.606679 26776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.585832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41524 (local address 127.15.228.126:36589)
0808 04:35:09.591700 (+ 5868us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.591713 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.591726 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:35:09.591762 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.591764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.591811 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.591919 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:09.594530 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.595372 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.597335 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.597772 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.597921 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.599674 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:09.599679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:09.599683 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:09.599713 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:09.603670 (+ 3957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.604211 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.604214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.604216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.604320 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.605444 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.605450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.605452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.605682 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:09.605817 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.606071 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.606198 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5788,"thread_start_us":91,"threads_started":1}
I20260808 04:35:09.607156 26761 heartbeater.cc:511] Master 127.15.228.126:36589 requested a full tablet report, sending...
I20260808 04:35:09.607180 26147 ts_manager.cc:194] Registered new tserver with Master: 2114de6f886e4314b4d47f30c9cea294 (127.15.228.67:38279)
I20260808 04:35:09.608078 25893 ts_manager.cc:194] Registered new tserver with Master: 2114de6f886e4314b4d47f30c9cea294 (127.15.228.67:38279)
I20260808 04:35:09.608165 25052 ts_manager.cc:194] Re-registered known tserver with Master: 2114de6f886e4314b4d47f30c9cea294 (127.15.228.67:38279)
I20260808 04:35:09.609005 25052 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:54701
I20260808 04:35:09.610193 26774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.585389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34506 (local address 127.15.228.125:42221)
0808 04:35:09.592135 (+ 6746us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.592141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.592154 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:35:09.592194 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.592200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.592267 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.592362 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:09.593490 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.594351 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.596739 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.597120 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.597223 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.605548 (+ 8325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:09.605552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:09.605554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:09.605578 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:09.608033 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.608644 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.608648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.608650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.608742 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.609040 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.609044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.609045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.609315 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:09.609450 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.609712 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.609842 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6671,"thread_start_us":90,"threads_started":1}
I20260808 04:35:09.610453 25052 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:35:09.613873 25052 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:35:09.616590 25838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7eb94118f70c4552bcab3f523c207610" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: NON_VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: true } } }
I20260808 04:35:09.616771 25838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:09.622136 26784 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610: attempting to promote NON_VOTER 1b5893d92b424d85bc9b7a9581005d68 to VOTER
I20260808 04:35:09.622553 26784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false } }
I20260808 04:35:09.623977 26159 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 1 LEARNER]: Refusing update from remote peer 7eb94118f70c4552bcab3f523c207610: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 04:35:09.624357 25905 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 1 FOLLOWER]: Refusing update from remote peer 7eb94118f70c4552bcab3f523c207610: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 04:35:09.624442 26783 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 04:35:09.624857 26784 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 04:35:09.626500 26784 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1b5893d92b424d85bc9b7a9581005d68 (127.15.228.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false } } }
I20260808 04:35:09.627475 26159 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1b5893d92b424d85bc9b7a9581005d68 (127.15.228.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false } } }
I20260808 04:35:09.628275 26784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7eb94118f70c4552bcab3f523c207610" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false } } }
I20260808 04:35:09.628377 26784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:09.628335 26095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7eb94118f70c4552bcab3f523c207610" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false } } }
I20260808 04:35:09.628453 26095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:09.628580 26784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7eb94118f70c4552bcab3f523c207610" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false } } }
I20260808 04:35:09.628690 26784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:09.629781 26201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7eb94118f70c4552bcab3f523c207610" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false } } }
I20260808 04:35:09.629909 26201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:35:09.630430 25905 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1b5893d92b424d85bc9b7a9581005d68 (127.15.228.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false } } }
I20260808 04:35:09.632423 26794 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7eb94118f70c4552bcab3f523c207610" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false } } }
I20260808 04:35:09.632663 26794 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:35:09.638108 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.631678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:38279 (local address 127.0.0.1:48528)
0808 04:35:09.631900 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:09.631913 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:09.631976 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:09.634089 (+ 2113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:09.634092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:09.634111 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:09.634231 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.634242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.635651 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.635655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.636468 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.636477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.637680 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.637684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.637772 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.637799 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 04:35:09.637898 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 04:35:09.637971 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260808 04:35:09.638268 26792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.631221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48528 (local address 127.15.228.67:38279)
0808 04:35:09.633569 (+ 2348us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.633582 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.633608 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:35:09.633698 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.633703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.633897 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.634124 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:09.634346 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.635549 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.636601 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.637437 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.637538 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.637758 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.637932 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.638113 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2222,"thread_start_us":150,"threads_started":1}
I20260808 04:35:09.639954 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.632799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:45007 (local address 127.0.0.1:60070)
0808 04:35:09.633463 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:09.633475 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:09.633551 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:09.635464 (+ 1913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:09.635467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:09.635485 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:09.635579 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.635588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.636488 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.636493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.638002 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.638013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.639532 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.639536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.639620 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.639645 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:35:09.639733 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 04:35:09.639829 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":1,"thread_start_us":131,"threads_started":1}
I20260808 04:35:09.640108 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.633419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60070 (local address 127.15.228.65:45007)
0808 04:35:09.635095 (+ 1676us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.635100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.635124 (+ 24us) server_negotiation.cc:407] Connection header received
0808 04:35:09.635187 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.635191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.635319 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.635487 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:09.635675 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.636373 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.638149 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.639398 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.639517 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.639650 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.639770 (+ 120us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.639868 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1581,"thread_start_us":95,"threads_started":1}
I20260808 04:35:09.640162 26796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.632801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:34177 (local address 127.0.0.1:54438)
0808 04:35:09.633133 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:09.633167 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:09.633276 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:09.634344 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:09.634347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:09.634363 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:09.634451 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.634461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.637004 (+ 2543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.637008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.638418 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.638428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.639721 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.639726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.639851 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.639873 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 04:35:09.639946 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 04:35:09.640006 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":20,"thread_start_us":147,"threads_started":1}
I20260808 04:35:09.640995 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.633278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54438 (local address 127.15.228.66:34177)
0808 04:35:09.633727 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.633743 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.633760 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:35:09.633823 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.633827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.634038 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.634191 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:09.634578 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.636680 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.638580 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.639436 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.639527 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.639650 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.640343 (+ 693us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.640440 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":128,"threads_started":1}
I20260808 04:35:09.641991 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 6cbc1b0ba0af4169b52cf71ae59cab72 (DEFAULT_TABLE table=test-table [id=f6daf679f90c4f5f9aa620da6470c485]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:09.642427 26297 tablet_service.cc:1467] Processing CreateTablet for tablet 6cbc1b0ba0af4169b52cf71ae59cab72 (DEFAULT_TABLE table=test-table [id=f6daf679f90c4f5f9aa620da6470c485]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:09.643342 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cbc1b0ba0af4169b52cf71ae59cab72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:09.643406 26297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cbc1b0ba0af4169b52cf71ae59cab72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:09.647261 26484 tablet_service.cc:1467] Processing CreateTablet for tablet 6cbc1b0ba0af4169b52cf71ae59cab72 (DEFAULT_TABLE table=test-table [id=f6daf679f90c4f5f9aa620da6470c485]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:09.648348 26484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cbc1b0ba0af4169b52cf71ae59cab72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:09.651705 26804 tablet_bootstrap.cc:492] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c: Bootstrap starting.
I20260808 04:35:09.653075 26805 tablet_bootstrap.cc:492] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294: Bootstrap starting.
I20260808 04:35:09.654284 26804 tablet_bootstrap.cc:654] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:09.655383 26805 tablet_bootstrap.cc:654] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:09.655851 26804 log.cc:824] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:09.656234 26806 tablet_bootstrap.cc:492] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9: Bootstrap starting.
I20260808 04:35:09.656450 26805 log.cc:824] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:09.658339 26806 tablet_bootstrap.cc:654] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:09.658538 26805 tablet_bootstrap.cc:492] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294: No bootstrap required, opened a new log
I20260808 04:35:09.658538 26804 tablet_bootstrap.cc:492] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c: No bootstrap required, opened a new log
I20260808 04:35:09.658728 26805 ts_tablet_manager.cc:1397] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:35:09.658733 26804 ts_tablet_manager.cc:1397] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 04:35:09.659322 26806 log.cc:824] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:09.661006 26806 tablet_bootstrap.cc:492] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9: No bootstrap required, opened a new log
I20260808 04:35:09.661185 26806 ts_tablet_manager.cc:1397] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:35:09.664234 26804 raft_consensus.cc:348] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 } } peers { permanent_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34177 } } peers { permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 } }
I20260808 04:35:09.664285 26805 raft_consensus.cc:348] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 } } peers { permanent_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34177 } } peers { permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 } }
I20260808 04:35:09.664521 26804 raft_consensus.cc:374] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:09.664521 26805 raft_consensus.cc:374] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:09.664580 26804 raft_consensus.cc:729] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eba8ed56d6047daa77802d8f0ffc71c, State: Initialized, Role: FOLLOWER
I20260808 04:35:09.664580 26805 raft_consensus.cc:729] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2114de6f886e4314b4d47f30c9cea294, State: Initialized, Role: FOLLOWER
I20260808 04:35:09.665113 26804 consensus_queue.cc:260] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 } } peers { permanent_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34177 } } peers { permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 } }
I20260808 04:35:09.665113 26805 consensus_queue.cc:260] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 } } peers { permanent_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34177 } } peers { permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 } }
I20260808 04:35:09.665481 26806 raft_consensus.cc:348] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 } } peers { permanent_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34177 } } peers { permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 } }
I20260808 04:35:09.665721 26806 raft_consensus.cc:374] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:09.665782 26806 raft_consensus.cc:729] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93fc6b49759b480a8cd03c65ecb21bb9, State: Initialized, Role: FOLLOWER
I20260808 04:35:09.665766 26385 heartbeater.cc:503] Master 127.15.228.126:36589 was elected leader, sending a full tablet report...
I20260808 04:35:09.666226 26805 ts_tablet_manager.cc:1428] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260808 04:35:09.666182 26806 consensus_queue.cc:260] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 } } peers { permanent_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34177 } } peers { permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 } }
I20260808 04:35:09.666605 26761 heartbeater.cc:503] Master 127.15.228.126:36589 was elected leader, sending a full tablet report...
I20260808 04:35:09.669138 26806 ts_tablet_manager.cc:1428] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9: Time spent starting tablet: real 0.008s user 0.001s sys 0.005s
I20260808 04:35:09.669590 26576 heartbeater.cc:503] Master 127.15.228.126:36589 was elected leader, sending a full tablet report...
I20260808 04:35:09.672792 26804 ts_tablet_manager.cc:1428] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c: Time spent starting tablet: real 0.014s user 0.004s sys 0.004s
I20260808 04:35:09.675132 26813 raft_consensus.cc:482] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:35:09.675289 26813 raft_consensus.cc:504] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 } } peers { permanent_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34177 } } peers { permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 } }
I20260808 04:35:09.677088 26813 leader_election.cc:288] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2114de6f886e4314b4d47f30c9cea294 (127.15.228.67:38279), 0eba8ed56d6047daa77802d8f0ffc71c (127.15.228.65:45007)
I20260808 04:35:09.681915 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.677411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:38987 (local address 127.15.228.65:45007)
0808 04:35:09.677541 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.677546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.677901 (+ 355us) server_negotiation.cc:407] Connection header received
0808 04:35:09.678121 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.678123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.678191 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.678274 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:09.678841 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.679544 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.680729 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.681260 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.681403 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.681455 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.681694 (+ 239us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.681786 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 04:35:09.681989 26580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.677428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:45007 (local address 127.15.228.66:38987)
0808 04:35:09.677768 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:09.677797 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:09.677973 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:09.678310 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:09.678313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:09.678333 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:09.678654 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.678666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.679699 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.679702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.680579 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.680588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.681395 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.681399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.681529 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.681559 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 04:35:09.681651 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 04:35:09.681730 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260808 04:35:09.682134 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.677374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:38279 (local address 127.15.228.66:54247)
0808 04:35:09.677693 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:09.677707 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:09.677826 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:09.678204 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:09.678210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:09.678232 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:09.678473 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.678481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.679541 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.679545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.680692 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:09.680706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.681645 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.681649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:09.681834 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.681856 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 04:35:09.681937 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 04:35:09.682027 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242}
I20260808 04:35:09.682857 26792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.677476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:54247 (local address 127.15.228.67:38279)
0808 04:35:09.677575 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.677578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.677776 (+ 198us) server_negotiation.cc:407] Connection header received
0808 04:35:09.678002 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.678008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.678085 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.678169 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:09.678630 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.679397 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.681018 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.681499 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.681689 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.681740 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.682667 (+ 927us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.682759 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 04:35:09.682824 26319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cbc1b0ba0af4169b52cf71ae59cab72" candidate_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" is_pre_election: true
I20260808 04:35:09.683166 26319 raft_consensus.cc:2454] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93fc6b49759b480a8cd03c65ecb21bb9 in term 0.
I20260808 04:35:09.683640 26695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cbc1b0ba0af4169b52cf71ae59cab72" candidate_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2114de6f886e4314b4d47f30c9cea294" is_pre_election: true
I20260808 04:35:09.683804 26422 leader_election.cc:302] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eba8ed56d6047daa77802d8f0ffc71c, 93fc6b49759b480a8cd03c65ecb21bb9; no voters:
I20260808 04:35:09.684016 26695 raft_consensus.cc:2454] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93fc6b49759b480a8cd03c65ecb21bb9 in term 0.
I20260808 04:35:09.684149 26813 raft_consensus.cc:2781] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:35:09.684222 26813 raft_consensus.cc:482] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:35:09.684266 26813 raft_consensus.cc:3037] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:09.685752 26813 raft_consensus.cc:504] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 } } peers { permanent_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34177 } } peers { permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 } }
I20260808 04:35:09.686061 26813 leader_election.cc:288] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [CANDIDATE]: Term 1 election: Requested vote from peers 2114de6f886e4314b4d47f30c9cea294 (127.15.228.67:38279), 0eba8ed56d6047daa77802d8f0ffc71c (127.15.228.65:45007)
I20260808 04:35:09.686583 26319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cbc1b0ba0af4169b52cf71ae59cab72" candidate_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eba8ed56d6047daa77802d8f0ffc71c"
I20260808 04:35:09.686719 26319 raft_consensus.cc:3037] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:09.687137 26695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cbc1b0ba0af4169b52cf71ae59cab72" candidate_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2114de6f886e4314b4d47f30c9cea294"
I20260808 04:35:09.687278 26695 raft_consensus.cc:3037] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:09.688268 26319 raft_consensus.cc:2454] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93fc6b49759b480a8cd03c65ecb21bb9 in term 1.
I20260808 04:35:09.688580 26422 leader_election.cc:302] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eba8ed56d6047daa77802d8f0ffc71c, 93fc6b49759b480a8cd03c65ecb21bb9; no voters:
I20260808 04:35:09.688751 26695 raft_consensus.cc:2454] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93fc6b49759b480a8cd03c65ecb21bb9 in term 1.
I20260808 04:35:09.688766 26813 raft_consensus.cc:2781] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:35:09.688879 26813 raft_consensus.cc:686] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 1 LEADER]: Becoming Leader. State: Replica: 93fc6b49759b480a8cd03c65ecb21bb9, State: Running, Role: LEADER
I20260808 04:35:09.689256 26813 consensus_queue.cc:237] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 } } peers { permanent_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34177 } } peers { permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 } }
I20260808 04:35:09.692672 25051 catalog_manager.cc:5495] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93fc6b49759b480a8cd03c65ecb21bb9 (127.15.228.66). New cstate: current_term: 1 leader_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 } health_report { overall_health: UNKNOWN } } }
I20260808 04:35:09.722458 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.717650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.15.228.66:34177)
0808 04:35:09.717784 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.717788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.717801 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:35:09.717880 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.717882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.717936 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.718030 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:35:09.718623 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.719526 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.720541 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.720765 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.720862 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.721243 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:35:09.721689 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:35:09.721792 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.722144 (+ 352us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.722280 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:35:09.729427 26695 raft_consensus.cc:1261] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294 [term 1 FOLLOWER]: Refusing update from remote peer 93fc6b49759b480a8cd03c65ecb21bb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:35:09.729427 26319 raft_consensus.cc:1261] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c [term 1 FOLLOWER]: Refusing update from remote peer 93fc6b49759b480a8cd03c65ecb21bb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:35:09.730329 26812 consensus_queue.cc:1035] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:35:09.730551 26813 consensus_queue.cc:1035] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:35:09.740464 26822 mvcc.cc:205] Tried to move back new op lower bound from 7316126555043868672 to 7316126554890440704. Current Snapshot: MvccSnapshot[applied={T|T < 7316126555043868672}]
I20260808 04:35:09.742268 26821 mvcc.cc:205] Tried to move back new op lower bound from 7316126555043868672 to 7316126554890440704. Current Snapshot: MvccSnapshot[applied={T|T < 7316126555043868672}]
I20260808 04:35:09.750866 25051 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:35:09.751096 25051 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:35:09.756039 25051 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7eb94118f70c4552bcab3f523c207610: Sending DeleteTablet for 3 replicas of tablet 6cbc1b0ba0af4169b52cf71ae59cab72
I20260808 04:35:09.757040 26668 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cbc1b0ba0af4169b52cf71ae59cab72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:09 UTC) from {username='oryx', principal='oryx/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:48528
I20260808 04:35:09.757180 26297 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cbc1b0ba0af4169b52cf71ae59cab72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:09 UTC) from {username='oryx', principal='oryx/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:60070
I20260808 04:35:09.757673 26831 tablet_replica.cc:330] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294: stopping tablet replica
I20260808 04:35:09.758081 26831 raft_consensus.cc:2229] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:35:09.758082 26832 tablet_replica.cc:330] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c: stopping tablet replica
I20260808 04:35:09.758384 26832 raft_consensus.cc:2229] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:35:09.758407 26484 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cbc1b0ba0af4169b52cf71ae59cab72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:09 UTC) from {username='oryx', principal='oryx/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:54438
I20260808 04:35:09.758661 26831 raft_consensus.cc:2258] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:09.758841 26832 raft_consensus.cc:2258] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:09.760519 26831 ts_tablet_manager.cc:1905] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:09.762198 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 25009
I20260808 04:35:09.762251 26832 ts_tablet_manager.cc:1905] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:09.776697 26831 ts_tablet_manager.cc:1918] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:35:09.777513 26831 log.cc:1196] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/6cbc1b0ba0af4169b52cf71ae59cab72
I20260808 04:35:09.777616 26832 ts_tablet_manager.cc:1918] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:35:09.777740 26832 log.cc:1196] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-0/wal/wals/6cbc1b0ba0af4169b52cf71ae59cab72
I20260808 04:35:09.778108 26832 ts_tablet_manager.cc:1939] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 0eba8ed56d6047daa77802d8f0ffc71c: Deleting consensus metadata
I20260808 04:35:09.778126 26831 ts_tablet_manager.cc:1939] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 2114de6f886e4314b4d47f30c9cea294: Deleting consensus metadata
I20260808 04:35:09.783414 26833 tablet_replica.cc:330] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9: stopping tablet replica
I20260808 04:35:09.784263 26833 raft_consensus.cc:2229] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:35:09.784967 26833 raft_consensus.cc:2258] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:09.787951 26833 ts_tablet_manager.cc:1905] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 04:35:09.788408 26242 connection.cc:535] server connection from 127.0.0.1:60070 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 04:35:09.788681 26611 connection.cc:535] server connection from 127.0.0.1:48528 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 04:35:09.792423 26833 ts_tablet_manager.cc:1918] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:35:09.792533 26833 log.cc:1196] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/6cbc1b0ba0af4169b52cf71ae59cab72
I20260808 04:35:09.792886 26833 ts_tablet_manager.cc:1939] T 6cbc1b0ba0af4169b52cf71ae59cab72 P 93fc6b49759b480a8cd03c65ecb21bb9: Deleting consensus metadata
W20260808 04:35:09.794176 26421 connection.cc:441] server connection from 127.0.0.1:54438 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54438 (request call id 1) could send its response
I20260808 04:35:09.797678 26774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.760817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34520 (local address 127.15.228.125:42221)
0808 04:35:09.761105 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.761112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.777272 (+ 16160us) server_negotiation.cc:407] Connection header received
0808 04:35:09.777452 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.777455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.777520 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.777608 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:09.784153 (+ 6545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.785040 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.786293 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.786652 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.786726 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.793097 (+ 6371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:09.793101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:09.793104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:09.793136 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:09.795100 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.795586 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.795590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.795592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.795676 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.795985 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.795989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.795990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.796200 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:09.796318 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.797381 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.797511 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260808 04:35:09.806825 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.793357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34536 (local address 127.15.228.125:42221)
0808 04:35:09.793475 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.793479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.794005 (+ 526us) server_negotiation.cc:407] Connection header received
0808 04:35:09.794427 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.794430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.794477 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.794580 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:09.795369 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.796017 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.797525 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.797847 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.797958 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.801574 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:09.801579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:09.801584 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:09.801631 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:09.803950 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.804577 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.804582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.804584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.804725 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.805087 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.805092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.805097 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.805584 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:09.805765 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.806499 (+ 734us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.806629 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:35:09.808382 26392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.796270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.15.228.124:36829)
0808 04:35:09.796432 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.796436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.798053 (+ 1617us) server_negotiation.cc:407] Connection header received
0808 04:35:09.798168 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.798171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.798216 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.798290 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:09.799257 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.800104 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.802003 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.802423 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.802588 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.804215 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:09.804221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:09.804224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:09.804254 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:09.806430 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.807099 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.807103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.807104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.807189 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.807473 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.807477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.807479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.807662 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:09.807807 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.808065 (+ 258us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.808175 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":82}
I20260808 04:35:09.811064 26779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:09.760929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42772 (local address 127.15.228.124:36829)
0808 04:35:09.761212 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:09.761218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:09.797048 (+ 35830us) server_negotiation.cc:407] Connection header received
0808 04:35:09.797102 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:09.797105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:09.797157 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:09.797227 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:09.800523 (+ 3296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.801314 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.802514 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:09.803146 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:09.803258 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:09.805934 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:09.805938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:09.805941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:09.805973 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:09.808784 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.809491 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.809495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.809497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.809579 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:09.810020 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:09.810024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:09.810026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:09.810227 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:09.810362 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:09.810622 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 04:35:09.810754 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":172,"server-negotiator.queue_time_us":102}
I20260808 04:35:10.745154 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:10.744615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.65:41497)
0808 04:35:10.744917 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:10.745029 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":122,"threads_started":1}
W20260808 04:35:10.745714 26385 heartbeater.cc:644] Failed to heartbeat to 127.15.228.126:36589 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
I20260808 04:35:10.746574 26846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:10.746054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.67:41173)
0808 04:35:10.746368 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:10.746464 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":73,"threads_started":1}
W20260808 04:35:10.746842 26761 heartbeater.cc:644] Failed to heartbeat to 127.15.228.126:36589 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
I20260808 04:35:10.746891 26847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:10.746332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.66:42213)
0808 04:35:10.746694 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:10.746788 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":80,"threads_started":1}
W20260808 04:35:10.747225 26576 heartbeater.cc:644] Failed to heartbeat to 127.15.228.126:36589 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
I20260808 04:35:11.256945 26848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7eb94118f70c4552bcab3f523c207610)
I20260808 04:35:11.257432 26848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false } }
I20260808 04:35:11.258136 26849 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7eb94118f70c4552bcab3f523c207610)
I20260808 04:35:11.258386 26849 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false } }
I20260808 04:35:11.259379 26849 leader_election.cc:288] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7eb94118f70c4552bcab3f523c207610 (127.15.228.126:36589), 1b5893d92b424d85bc9b7a9581005d68 (127.15.228.124:36829)
I20260808 04:35:11.259876 26848 leader_election.cc:288] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7eb94118f70c4552bcab3f523c207610 (127.15.228.126:36589), fd4721373d7b4a7686b043b6116850eb (127.15.228.125:42221)
I20260808 04:35:11.260186 26850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:11.259542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.0.0.1:42000)
0808 04:35:11.259956 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:11.260081 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":78,"threads_started":1}
W20260808 04:35:11.260401 26128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
I20260808 04:35:11.260551 26855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:11.259605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.0.0.1:42012)
0808 04:35:11.260370 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:11.260453 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":79,"threads_started":1}
W20260808 04:35:11.260717 25871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
I20260808 04:35:11.261847 26855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:11.261596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.0.0.1:42028)
0808 04:35:11.261734 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:11.261781 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 04:35:11.261847 26850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:11.261596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.0.0.1:42026)
0808 04:35:11.261734 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:11.261781 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260808 04:35:11.262002 25871 leader_election.cc:334] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7eb94118f70c4552bcab3f523c207610 (127.15.228.126:36589): Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
W20260808 04:35:11.262071 26128 leader_election.cc:334] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7eb94118f70c4552bcab3f523c207610 (127.15.228.126:36589): Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
I20260808 04:35:11.265751 26851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:11.260022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36829 (local address 127.0.0.1:42792)
0808 04:35:11.260523 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:11.260542 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:11.260798 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:11.261448 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:11.261456 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:11.261479 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:11.261582 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:11.261590 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:11.262883 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:11.262887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:11.264208 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:11.264219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:11.265304 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:11.265309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:11.265415 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:11.265443 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:35:11.265533 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 04:35:11.265618 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":92,"threads_started":1}
I20260808 04:35:11.265753 26852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:11.259683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:42221 (local address 127.0.0.1:34550)
0808 04:35:11.260637 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:11.260652 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:11.260920 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:11.261363 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:11.261366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:11.261455 (+ 89us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:11.261582 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:11.261662 (+ 80us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:11.262876 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:11.262879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:11.264208 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:11.264219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:11.265293 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:11.265297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:11.265456 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:11.265487 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 04:35:11.265564 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 04:35:11.265631 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":42,"thread_start_us":76,"threads_started":1}
I20260808 04:35:11.265901 26857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:11.260230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42792 (local address 127.15.228.124:36829)
0808 04:35:11.261141 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:11.261145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:11.261159 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:35:11.261205 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:11.261208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:11.261244 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:11.261713 (+ 469us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:11.261813 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:11.262658 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:11.264370 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:11.265126 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:11.265262 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:11.265369 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:11.265625 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 04:35:11.265718 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":833,"thread_start_us":101,"threads_started":1}
I20260808 04:35:11.265911 26853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:11.259799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.15.228.125:42221)
0808 04:35:11.260198 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:11.260206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:11.260827 (+ 621us) server_negotiation.cc:407] Connection header received
0808 04:35:11.261083 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:11.261087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:11.261137 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:11.261200 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:11.261787 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:11.262607 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:11.264369 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:11.265127 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:11.265262 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:11.265320 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:11.265625 (+ 305us) server_negotiation.cc:299] Negotiation successful
0808 04:35:11.265708 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":70,"threads_started":1}
I20260808 04:35:11.266743 26159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd4721373d7b4a7686b043b6116850eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1b5893d92b424d85bc9b7a9581005d68" is_pre_election: true
I20260808 04:35:11.266806 25905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b5893d92b424d85bc9b7a9581005d68" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fd4721373d7b4a7686b043b6116850eb" is_pre_election: true
I20260808 04:35:11.267031 26159 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd4721373d7b4a7686b043b6116850eb in term 1.
I20260808 04:35:11.267035 25905 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b5893d92b424d85bc9b7a9581005d68 in term 1.
I20260808 04:35:11.267416 25871 leader_election.cc:302] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b5893d92b424d85bc9b7a9581005d68, fd4721373d7b4a7686b043b6116850eb; no voters: 7eb94118f70c4552bcab3f523c207610
I20260808 04:35:11.267432 26129 leader_election.cc:302] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b5893d92b424d85bc9b7a9581005d68, fd4721373d7b4a7686b043b6116850eb; no voters: 7eb94118f70c4552bcab3f523c207610
I20260808 04:35:11.267648 26849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 04:35:11.267668 26848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 04:35:11.267724 26848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7eb94118f70c4552bcab3f523c207610)
I20260808 04:35:11.267719 26849 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7eb94118f70c4552bcab3f523c207610)
I20260808 04:35:11.267763 26848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:35:11.267767 26849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:35:11.269183 26849 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false } }
I20260808 04:35:11.269243 26848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false } }
I20260808 04:35:11.269500 26849 leader_election.cc:288] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [CANDIDATE]: Term 2 election: Requested vote from peers 7eb94118f70c4552bcab3f523c207610 (127.15.228.126:36589), 1b5893d92b424d85bc9b7a9581005d68 (127.15.228.124:36829)
I20260808 04:35:11.269650 26848 leader_election.cc:288] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [CANDIDATE]: Term 2 election: Requested vote from peers 7eb94118f70c4552bcab3f523c207610 (127.15.228.126:36589), fd4721373d7b4a7686b043b6116850eb (127.15.228.125:42221)
I20260808 04:35:11.270018 26852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:11.269742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.0.0.1:42058)
0808 04:35:11.269851 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:11.269945 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260808 04:35:11.270104 26851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:11.269677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.0.0.1:42056)
0808 04:35:11.269949 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:11.270005 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206}
I20260808 04:35:11.270179 25905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b5893d92b424d85bc9b7a9581005d68" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fd4721373d7b4a7686b043b6116850eb"
I20260808 04:35:11.270345 25905 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b5893d92b424d85bc9b7a9581005d68 in current term 2: Already voted for candidate fd4721373d7b4a7686b043b6116850eb in this term.
I20260808 04:35:11.270726 26159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd4721373d7b4a7686b043b6116850eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1b5893d92b424d85bc9b7a9581005d68"
I20260808 04:35:11.270802 26851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:11.270599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.0.0.1:42068)
0808 04:35:11.270688 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:11.270737 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260808 04:35:11.270896 26159 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fd4721373d7b4a7686b043b6116850eb in current term 2: Already voted for candidate 1b5893d92b424d85bc9b7a9581005d68 in this term.
W20260808 04:35:11.270902 25871 leader_election.cc:334] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7eb94118f70c4552bcab3f523c207610 (127.15.228.126:36589): Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
I20260808 04:35:11.271236 25871 leader_election.cc:302] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd4721373d7b4a7686b043b6116850eb; no voters: 1b5893d92b424d85bc9b7a9581005d68, 7eb94118f70c4552bcab3f523c207610
I20260808 04:35:11.271421 26849 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 04:35:11.271575 26852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:11.271383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.0.0.1:42080)
0808 04:35:11.271473 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:11.271520 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260808 04:35:11.271677 26128 leader_election.cc:334] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7eb94118f70c4552bcab3f523c207610 (127.15.228.126:36589): Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
I20260808 04:35:11.271771 26128 leader_election.cc:302] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b5893d92b424d85bc9b7a9581005d68; no voters: 7eb94118f70c4552bcab3f523c207610, fd4721373d7b4a7686b043b6116850eb
I20260808 04:35:11.271888 26848 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 04:35:11.747152 26860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:11.746494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.65:39655)
0808 04:35:11.746891 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:11.747013 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":105,"threads_started":1}
I20260808 04:35:11.747996 26861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:11.747372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.67:47781)
0808 04:35:11.747836 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:11.747898 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":78,"threads_started":1}
I20260808 04:35:11.748636 26863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:11.747936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.66:35079)
0808 04:35:11.748454 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:11.748541 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":84,"threads_started":1}
I20260808 04:35:12.749192 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:12.748298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.65:52963)
0808 04:35:12.748882 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:12.748979 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":94,"threads_started":1}
I20260808 04:35:12.751106 26866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:12.749740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.66:56545)
0808 04:35:12.750681 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:12.750865 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":82,"threads_started":1}
I20260808 04:35:12.751338 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:12.748973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.67:41519)
0808 04:35:12.750122 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:12.750201 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1039,"thread_start_us":103,"threads_started":1}
I20260808 04:35:13.064275 26867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:35:13.064670 26867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false } }
I20260808 04:35:13.065423 26867 leader_election.cc:288] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7eb94118f70c4552bcab3f523c207610 (127.15.228.126:36589), fd4721373d7b4a7686b043b6116850eb (127.15.228.125:42221)
I20260808 04:35:13.066208 26868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.065687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.0.0.1:42360)
0808 04:35:13.066019 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.066113 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":112,"threads_started":1}
I20260808 04:35:13.066212 25905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b5893d92b424d85bc9b7a9581005d68" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fd4721373d7b4a7686b043b6116850eb" is_pre_election: true
I20260808 04:35:13.066442 25905 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b5893d92b424d85bc9b7a9581005d68 in term 2.
I20260808 04:35:13.066808 26129 leader_election.cc:302] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b5893d92b424d85bc9b7a9581005d68, fd4721373d7b4a7686b043b6116850eb; no voters:
I20260808 04:35:13.067046 26867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 04:35:13.067123 26867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:35:13.067168 26867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 2 FOLLOWER]: Advancing to term 3
I20260808 04:35:13.067376 26868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.067106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.0.0.1:42364)
0808 04:35:13.067262 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.067322 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260808 04:35:13.067523 26128 leader_election.cc:334] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7eb94118f70c4552bcab3f523c207610 (127.15.228.126:36589): Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
I20260808 04:35:13.068557 26867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false } }
I20260808 04:35:13.069025 26867 leader_election.cc:288] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [CANDIDATE]: Term 3 election: Requested vote from peers 7eb94118f70c4552bcab3f523c207610 (127.15.228.126:36589), fd4721373d7b4a7686b043b6116850eb (127.15.228.125:42221)
I20260808 04:35:13.069443 26868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.069109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.0.0.1:42378)
0808 04:35:13.069337 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.069393 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260808 04:35:13.069557 25905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b5893d92b424d85bc9b7a9581005d68" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fd4721373d7b4a7686b043b6116850eb"
I20260808 04:35:13.069777 25905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 2 FOLLOWER]: Advancing to term 3
I20260808 04:35:13.070586 26868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.069999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.0.0.1:42398)
0808 04:35:13.070265 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.070448 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 04:35:13.070832 26128 leader_election.cc:334] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7eb94118f70c4552bcab3f523c207610 (127.15.228.126:36589): Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
I20260808 04:35:13.071174 25905 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b5893d92b424d85bc9b7a9581005d68 in term 3.
I20260808 04:35:13.071516 26129 leader_election.cc:302] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b5893d92b424d85bc9b7a9581005d68, fd4721373d7b4a7686b043b6116850eb; no voters: 7eb94118f70c4552bcab3f523c207610
I20260808 04:35:13.071758 26867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 04:35:13.072057 26867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [term 3 LEADER]: Becoming Leader. State: Replica: 1b5893d92b424d85bc9b7a9581005d68, State: Running, Role: LEADER
I20260808 04:35:13.072463 26867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false } }
I20260808 04:35:13.074489 26871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b5893d92b424d85bc9b7a9581005d68. Latest consensus state: current_term: 3 leader_uuid: "1b5893d92b424d85bc9b7a9581005d68" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false } } }
I20260808 04:35:13.074671 26871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:13.075179 26873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:35:13.077673 26873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:35:13.078066 26873 catalog_manager.cc:1259] Loaded cluster ID: 7b0fb3629a754d6186ff6c61f948734c
I20260808 04:35:13.078117 26873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:35:13.078637 26873 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:35:13.079069 26873 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: Loaded TSK: 0
I20260808 04:35:13.079738 26873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:35:13.162050 26147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:35:13.166849 26147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:35:13.170255 26868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.169940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.0.0.1:42412)
0808 04:35:13.170073 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.170162 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 04:35:13.170380 25905 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [term 3 FOLLOWER]: Refusing update from remote peer 1b5893d92b424d85bc9b7a9581005d68: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260808 04:35:13.171175 26867 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 04:35:13.171420 26868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.171122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.0.0.1:42420)
0808 04:35:13.171254 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.171310 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 04:35:13.171615 26128 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 -> Peer 7eb94118f70c4552bcab3f523c207610 (127.15.228.126:36589): Couldn't send request to peer 7eb94118f70c4552bcab3f523c207610. Status: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 04:35:13.172812 26875 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b5893d92b424d85bc9b7a9581005d68. Latest consensus state: current_term: 3 leader_uuid: "1b5893d92b424d85bc9b7a9581005d68" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false } } }
I20260808 04:35:13.172955 26875 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:35:13.174002 26874 mvcc.cc:205] Tried to move back new op lower bound from 7316126569140277248 to 7316126568748625920. Current Snapshot: MvccSnapshot[applied={T|T < 7316126569140277248}]
I20260808 04:35:13.174249 26871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "1b5893d92b424d85bc9b7a9581005d68" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false } } }
I20260808 04:35:13.174378 26871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:13.174481 26877 mvcc.cc:205] Tried to move back new op lower bound from 7316126569140277248 to 7316126568748625920. Current Snapshot: MvccSnapshot[applied={T|T < 7316126569140277248}]
I20260808 04:35:13.174887 26875 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "1b5893d92b424d85bc9b7a9581005d68" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7eb94118f70c4552bcab3f523c207610" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36589 } } peers { permanent_uuid: "fd4721373d7b4a7686b043b6116850eb" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 42221 } attrs { promote: false } } peers { permanent_uuid: "1b5893d92b424d85bc9b7a9581005d68" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36829 } attrs { promote: false } } }
I20260808 04:35:13.175046 26875 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4721373d7b4a7686b043b6116850eb [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:35:13.186528 26884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.181978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:45007 (local address 127.0.0.1:60078)
0808 04:35:13.183089 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.183103 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:13.183213 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:13.183587 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:13.183589 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:13.183608 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:13.183690 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:13.183695 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:13.184580 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:13.184584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:13.185374 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:13.185383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:13.186044 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:13.186048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:13.186138 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:13.186165 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 04:35:13.186259 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 04:35:13.186356 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1051,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
I20260808 04:35:13.186688 26868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.181512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:34177 (local address 127.0.0.1:54446)
0808 04:35:13.181831 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.181846 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:13.181940 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:13.182894 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:13.182897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:13.182920 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:13.183100 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:13.183111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:13.184227 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:13.184230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:13.185379 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:13.185389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:13.186268 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:13.186271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:13.186381 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:13.186402 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 04:35:13.186476 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 04:35:13.186539 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":55}
I20260808 04:35:13.186751 26883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.181633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54446 (local address 127.15.228.66:34177)
0808 04:35:13.182654 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:13.182658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:13.182671 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:35:13.182731 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:13.182733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:13.182773 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:13.182848 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:13.183282 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:13.184105 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:13.185516 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:13.186140 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:13.186246 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:13.186306 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:13.186509 (+ 203us) server_negotiation.cc:299] Negotiation successful
0808 04:35:13.186589 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":944,"thread_start_us":84,"threads_started":1}
I20260808 04:35:13.187058 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.182332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60078 (local address 127.15.228.65:45007)
0808 04:35:13.182600 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:13.182605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:13.183249 (+ 644us) server_negotiation.cc:407] Connection header received
0808 04:35:13.183311 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:13.183314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:13.183365 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:13.183460 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:13.183816 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:13.184467 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:13.185502 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:13.185914 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:13.186033 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:13.186104 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:13.186784 (+ 680us) server_negotiation.cc:299] Negotiation successful
0808 04:35:13.186882 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":114,"threads_started":1}
I20260808 04:35:13.187345 26484 tablet_service.cc:1467] Processing CreateTablet for tablet 526077d6c83f4b44be6ad82526d62a46 (DEFAULT_TABLE table=test-table [id=eeb8ce94f85e4a878ca24dfd177d9e83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:13.187608 26484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526077d6c83f4b44be6ad82526d62a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:13.188458 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.181994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:38279 (local address 127.0.0.1:48534)
0808 04:35:13.183368 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.183377 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:13.183487 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:13.183729 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:13.183732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:13.183748 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:13.183842 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:13.183847 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:13.185514 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:13.185517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:13.187211 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:13.187222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:13.188075 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:13.188078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:13.188161 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:13.188191 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 04:35:13.188276 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 04:35:13.188341 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1330,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260808 04:35:13.188584 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.182351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48534 (local address 127.15.228.67:38279)
0808 04:35:13.182613 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:13.182618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:13.183462 (+ 844us) server_negotiation.cc:407] Connection header received
0808 04:35:13.183573 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:13.183576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:13.183624 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:13.183722 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:13.183960 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:13.185383 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:13.187313 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:13.187940 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:13.188049 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:13.188112 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:13.188317 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 04:35:13.188425 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":115,"threads_started":1}
I20260808 04:35:13.190709 26888 tablet_bootstrap.cc:492] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9: Bootstrap starting.
I20260808 04:35:13.191277 26297 tablet_service.cc:1467] Processing CreateTablet for tablet 526077d6c83f4b44be6ad82526d62a46 (DEFAULT_TABLE table=test-table [id=eeb8ce94f85e4a878ca24dfd177d9e83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:13.191577 26297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526077d6c83f4b44be6ad82526d62a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:13.191849 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 526077d6c83f4b44be6ad82526d62a46 (DEFAULT_TABLE table=test-table [id=eeb8ce94f85e4a878ca24dfd177d9e83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:13.192287 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526077d6c83f4b44be6ad82526d62a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:13.192361 26888 tablet_bootstrap.cc:654] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:13.193943 26888 tablet_bootstrap.cc:492] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9: No bootstrap required, opened a new log
I20260808 04:35:13.194031 26888 ts_tablet_manager.cc:1397] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 04:35:13.194468 26890 tablet_bootstrap.cc:492] T 526077d6c83f4b44be6ad82526d62a46 P 0eba8ed56d6047daa77802d8f0ffc71c: Bootstrap starting.
I20260808 04:35:13.194514 26888 raft_consensus.cc:348] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34177 } } peers { permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 } } peers { permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 } }
I20260808 04:35:13.194651 26888 raft_consensus.cc:374] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:13.194690 26888 raft_consensus.cc:729] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93fc6b49759b480a8cd03c65ecb21bb9, State: Initialized, Role: FOLLOWER
I20260808 04:35:13.194839 26888 consensus_queue.cc:260] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34177 } } peers { permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 } } peers { permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 } }
I20260808 04:35:13.196107 26866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.195746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.66:50423)
0808 04:35:13.195970 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.196031 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260808 04:35:13.196261 26893 tablet_bootstrap.cc:492] T 526077d6c83f4b44be6ad82526d62a46 P 2114de6f886e4314b4d47f30c9cea294: Bootstrap starting.
I20260808 04:35:13.196310 26890 tablet_bootstrap.cc:654] T 526077d6c83f4b44be6ad82526d62a46 P 0eba8ed56d6047daa77802d8f0ffc71c: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:13.197476 26893 tablet_bootstrap.cc:654] T 526077d6c83f4b44be6ad82526d62a46 P 2114de6f886e4314b4d47f30c9cea294: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:13.198177 26890 tablet_bootstrap.cc:492] T 526077d6c83f4b44be6ad82526d62a46 P 0eba8ed56d6047daa77802d8f0ffc71c: No bootstrap required, opened a new log
I20260808 04:35:13.198318 26890 ts_tablet_manager.cc:1397] T 526077d6c83f4b44be6ad82526d62a46 P 0eba8ed56d6047daa77802d8f0ffc71c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 04:35:13.199018 26890 raft_consensus.cc:348] T 526077d6c83f4b44be6ad82526d62a46 P 0eba8ed56d6047daa77802d8f0ffc71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34177 } } peers { permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 } } peers { permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 } }
I20260808 04:35:13.199184 26890 raft_consensus.cc:374] T 526077d6c83f4b44be6ad82526d62a46 P 0eba8ed56d6047daa77802d8f0ffc71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:13.199199 26893 tablet_bootstrap.cc:492] T 526077d6c83f4b44be6ad82526d62a46 P 2114de6f886e4314b4d47f30c9cea294: No bootstrap required, opened a new log
I20260808 04:35:13.199226 26890 raft_consensus.cc:729] T 526077d6c83f4b44be6ad82526d62a46 P 0eba8ed56d6047daa77802d8f0ffc71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eba8ed56d6047daa77802d8f0ffc71c, State: Initialized, Role: FOLLOWER
I20260808 04:35:13.199214 26888 ts_tablet_manager.cc:1428] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260808 04:35:13.199296 26893 ts_tablet_manager.cc:1397] T 526077d6c83f4b44be6ad82526d62a46 P 2114de6f886e4314b4d47f30c9cea294: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:35:13.199436 26575 heartbeater.cc:503] Master 127.15.228.124:36829 was elected leader, sending a full tablet report...
I20260808 04:35:13.199432 26890 consensus_queue.cc:260] T 526077d6c83f4b44be6ad82526d62a46 P 0eba8ed56d6047daa77802d8f0ffc71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34177 } } peers { permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 } } peers { permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 } }
I20260808 04:35:13.199870 26890 ts_tablet_manager.cc:1428] T 526077d6c83f4b44be6ad82526d62a46 P 0eba8ed56d6047daa77802d8f0ffc71c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260808 04:35:13.199869 26893 raft_consensus.cc:348] T 526077d6c83f4b44be6ad82526d62a46 P 2114de6f886e4314b4d47f30c9cea294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34177 } } peers { permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 } } peers { permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 } }
I20260808 04:35:13.199991 26893 raft_consensus.cc:374] T 526077d6c83f4b44be6ad82526d62a46 P 2114de6f886e4314b4d47f30c9cea294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:13.200035 26893 raft_consensus.cc:729] T 526077d6c83f4b44be6ad82526d62a46 P 2114de6f886e4314b4d47f30c9cea294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2114de6f886e4314b4d47f30c9cea294, State: Initialized, Role: FOLLOWER
I20260808 04:35:13.200332 26893 consensus_queue.cc:260] T 526077d6c83f4b44be6ad82526d62a46 P 2114de6f886e4314b4d47f30c9cea294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34177 } } peers { permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 } } peers { permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 } }
I20260808 04:35:13.200623 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.200337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.65:48253)
0808 04:35:13.200478 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.200539 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 04:35:13.200918 26893 ts_tablet_manager.cc:1428] T 526077d6c83f4b44be6ad82526d62a46 P 2114de6f886e4314b4d47f30c9cea294: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 04:35:13.201776 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.201519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.67:42201)
0808 04:35:13.201645 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.201705 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 04:35:13.202451 26384 heartbeater.cc:503] Master 127.15.228.124:36829 was elected leader, sending a full tablet report...
I20260808 04:35:13.203078 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.202843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.65:33915)
0808 04:35:13.202960 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.203012 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260808 04:35:13.203236 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.202514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.67:46263)
0808 04:35:13.203107 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.203170 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448}
I20260808 04:35:13.205533 26866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.205265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.66:48571)
0808 04:35:13.205407 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.205463 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 04:35:13.259300 26892 raft_consensus.cc:482] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:35:13.259473 26892 raft_consensus.cc:504] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34177 } } peers { permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 } } peers { permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 } }
I20260808 04:35:13.259949 26892 leader_election.cc:288] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0eba8ed56d6047daa77802d8f0ffc71c (127.15.228.65:45007), 2114de6f886e4314b4d47f30c9cea294 (127.15.228.67:38279)
I20260808 04:35:13.260558 26319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526077d6c83f4b44be6ad82526d62a46" candidate_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" is_pre_election: true
I20260808 04:35:13.260602 26695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526077d6c83f4b44be6ad82526d62a46" candidate_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2114de6f886e4314b4d47f30c9cea294" is_pre_election: true
I20260808 04:35:13.260749 26319 raft_consensus.cc:2454] T 526077d6c83f4b44be6ad82526d62a46 P 0eba8ed56d6047daa77802d8f0ffc71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93fc6b49759b480a8cd03c65ecb21bb9 in term 0.
I20260808 04:35:13.260749 26695 raft_consensus.cc:2454] T 526077d6c83f4b44be6ad82526d62a46 P 2114de6f886e4314b4d47f30c9cea294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93fc6b49759b480a8cd03c65ecb21bb9 in term 0.
I20260808 04:35:13.261101 26422 leader_election.cc:302] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eba8ed56d6047daa77802d8f0ffc71c, 93fc6b49759b480a8cd03c65ecb21bb9; no voters:
I20260808 04:35:13.261298 26892 raft_consensus.cc:2781] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:35:13.261369 26892 raft_consensus.cc:482] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:35:13.261399 26892 raft_consensus.cc:3037] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:13.262637 26892 raft_consensus.cc:504] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34177 } } peers { permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 } } peers { permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 } }
I20260808 04:35:13.263000 26892 leader_election.cc:288] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [CANDIDATE]: Term 1 election: Requested vote from peers 0eba8ed56d6047daa77802d8f0ffc71c (127.15.228.65:45007), 2114de6f886e4314b4d47f30c9cea294 (127.15.228.67:38279)
I20260808 04:35:13.263353 26695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526077d6c83f4b44be6ad82526d62a46" candidate_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2114de6f886e4314b4d47f30c9cea294"
I20260808 04:35:13.263355 26319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526077d6c83f4b44be6ad82526d62a46" candidate_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eba8ed56d6047daa77802d8f0ffc71c"
I20260808 04:35:13.263495 26695 raft_consensus.cc:3037] T 526077d6c83f4b44be6ad82526d62a46 P 2114de6f886e4314b4d47f30c9cea294 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:13.263505 26319 raft_consensus.cc:3037] T 526077d6c83f4b44be6ad82526d62a46 P 0eba8ed56d6047daa77802d8f0ffc71c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:13.264458 26319 raft_consensus.cc:2454] T 526077d6c83f4b44be6ad82526d62a46 P 0eba8ed56d6047daa77802d8f0ffc71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93fc6b49759b480a8cd03c65ecb21bb9 in term 1.
I20260808 04:35:13.264477 26695 raft_consensus.cc:2454] T 526077d6c83f4b44be6ad82526d62a46 P 2114de6f886e4314b4d47f30c9cea294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93fc6b49759b480a8cd03c65ecb21bb9 in term 1.
I20260808 04:35:13.264767 26422 leader_election.cc:302] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eba8ed56d6047daa77802d8f0ffc71c, 93fc6b49759b480a8cd03c65ecb21bb9; no voters:
I20260808 04:35:13.264952 26892 raft_consensus.cc:2781] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:35:13.265153 26892 raft_consensus.cc:686] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 1 LEADER]: Becoming Leader. State: Replica: 93fc6b49759b480a8cd03c65ecb21bb9, State: Running, Role: LEADER
I20260808 04:35:13.265273 26892 consensus_queue.cc:237] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34177 } } peers { permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 } } peers { permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 } }
I20260808 04:35:13.266352 26866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.266042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.66:37565)
0808 04:35:13.266206 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.266267 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260808 04:35:13.267366 26147 catalog_manager.cc:5495] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93fc6b49759b480a8cd03c65ecb21bb9 (127.15.228.66). New cstate: current_term: 1 leader_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93fc6b49759b480a8cd03c65ecb21bb9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 } health_report { overall_health: UNKNOWN } } }
I20260808 04:35:13.289570 26883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.285135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54462 (local address 127.15.228.66:34177)
0808 04:35:13.285265 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:13.285269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:13.285282 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:35:13.285376 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:13.285378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:13.285430 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:13.285496 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:35:13.285911 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:13.286572 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:13.287955 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:13.288196 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:13.288294 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:13.288670 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:35:13.288797 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:35:13.288896 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:13.289178 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 04:35:13.289320 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260808 04:35:13.289577 26391 tablet.cc:2369] T 526077d6c83f4b44be6ad82526d62a46 P 0eba8ed56d6047daa77802d8f0ffc71c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:35:13.293464 26319 raft_consensus.cc:1261] T 526077d6c83f4b44be6ad82526d62a46 P 0eba8ed56d6047daa77802d8f0ffc71c [term 1 FOLLOWER]: Refusing update from remote peer 93fc6b49759b480a8cd03c65ecb21bb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:35:13.293479 26695 raft_consensus.cc:1261] T 526077d6c83f4b44be6ad82526d62a46 P 2114de6f886e4314b4d47f30c9cea294 [term 1 FOLLOWER]: Refusing update from remote peer 93fc6b49759b480a8cd03c65ecb21bb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:35:13.294210 26900 consensus_queue.cc:1035] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eba8ed56d6047daa77802d8f0ffc71c" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:35:13.294450 26892 consensus_queue.cc:1035] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2114de6f886e4314b4d47f30c9cea294" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 38279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:35:13.295918 26759 heartbeater.cc:503] Master 127.15.228.124:36829 was elected leader, sending a full tablet report...
I20260808 04:35:13.297348 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.296691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.67:34029)
0808 04:35:13.296863 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.296932 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260808 04:35:13.297848 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.297582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.65:52193)
0808 04:35:13.297699 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.297757 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 04:35:13.302033 26866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.301670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.66:48015)
0808 04:35:13.301883 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.301942 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260808 04:35:13.302095 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.301649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.67:47803)
0808 04:35:13.301909 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.301978 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260808 04:35:13.303551 26866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.302974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.66:43533)
0808 04:35:13.303381 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.303468 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318}
I20260808 04:35:13.305343 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.305045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36589 (local address 127.15.228.65:57011)
0808 04:35:13.305194 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.305254 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:36589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260808 04:35:13.311062 26145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:35:13.311220 26145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:35:13.316903 26145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b5893d92b424d85bc9b7a9581005d68: Sending DeleteTablet for 3 replicas of tablet 526077d6c83f4b44be6ad82526d62a46
I20260808 04:35:13.317902 26484 tablet_service.cc:1514] Processing DeleteTablet for tablet 526077d6c83f4b44be6ad82526d62a46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:13 UTC) from {username='oryx', principal='oryx/127.15.228.124@KRBTEST.COM'} at 127.0.0.1:54446
I20260808 04:35:13.317943 26668 tablet_service.cc:1514] Processing DeleteTablet for tablet 526077d6c83f4b44be6ad82526d62a46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:13 UTC) from {username='oryx', principal='oryx/127.15.228.124@KRBTEST.COM'} at 127.0.0.1:48534
I20260808 04:35:13.317967 26297 tablet_service.cc:1514] Processing DeleteTablet for tablet 526077d6c83f4b44be6ad82526d62a46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:13 UTC) from {username='oryx', principal='oryx/127.15.228.124@KRBTEST.COM'} at 127.0.0.1:60078
I20260808 04:35:13.318471 26920 tablet_replica.cc:330] T 526077d6c83f4b44be6ad82526d62a46 P 0eba8ed56d6047daa77802d8f0ffc71c: stopping tablet replica
I20260808 04:35:13.318615 26920 raft_consensus.cc:2229] T 526077d6c83f4b44be6ad82526d62a46 P 0eba8ed56d6047daa77802d8f0ffc71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:35:13.318768 26920 raft_consensus.cc:2258] T 526077d6c83f4b44be6ad82526d62a46 P 0eba8ed56d6047daa77802d8f0ffc71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:13.319566 26920 ts_tablet_manager.cc:1905] T 526077d6c83f4b44be6ad82526d62a46 P 0eba8ed56d6047daa77802d8f0ffc71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:13.319670 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 26094
I20260808 04:35:13.319622 26919 tablet_replica.cc:330] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9: stopping tablet replica
I20260808 04:35:13.319947 26919 raft_consensus.cc:2229] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:35:13.319245 26918 tablet_replica.cc:330] T 526077d6c83f4b44be6ad82526d62a46 P 2114de6f886e4314b4d47f30c9cea294: stopping tablet replica
I20260808 04:35:13.320353 26918 raft_consensus.cc:2229] T 526077d6c83f4b44be6ad82526d62a46 P 2114de6f886e4314b4d47f30c9cea294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:35:13.320703 26919 raft_consensus.cc:2258] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:13.320974 26918 raft_consensus.cc:2258] T 526077d6c83f4b44be6ad82526d62a46 P 2114de6f886e4314b4d47f30c9cea294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:13.327720 26918 ts_tablet_manager.cc:1905] T 526077d6c83f4b44be6ad82526d62a46 P 2114de6f886e4314b4d47f30c9cea294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:13.336030 26918 ts_tablet_manager.cc:1918] T 526077d6c83f4b44be6ad82526d62a46 P 2114de6f886e4314b4d47f30c9cea294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:35:13.336166 26918 log.cc:1196] T 526077d6c83f4b44be6ad82526d62a46 P 2114de6f886e4314b4d47f30c9cea294: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/526077d6c83f4b44be6ad82526d62a46
I20260808 04:35:13.336486 26918 ts_tablet_manager.cc:1939] T 526077d6c83f4b44be6ad82526d62a46 P 2114de6f886e4314b4d47f30c9cea294: Deleting consensus metadata
I20260808 04:35:13.337592 26919 ts_tablet_manager.cc:1905] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:13.337679 26132 catalog_manager.cc:4841] TS 2114de6f886e4314b4d47f30c9cea294 (127.15.228.67:38279): tablet 526077d6c83f4b44be6ad82526d62a46 (table test-table [id=eeb8ce94f85e4a878ca24dfd177d9e83]) successfully deleted
I20260808 04:35:13.341967 26919 ts_tablet_manager.cc:1918] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:35:13.342080 26919 log.cc:1196] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/526077d6c83f4b44be6ad82526d62a46
I20260808 04:35:13.342446 26919 ts_tablet_manager.cc:1939] T 526077d6c83f4b44be6ad82526d62a46 P 93fc6b49759b480a8cd03c65ecb21bb9: Deleting consensus metadata
I20260808 04:35:13.343724 26128 catalog_manager.cc:4841] TS 93fc6b49759b480a8cd03c65ecb21bb9 (127.15.228.66:34177): tablet 526077d6c83f4b44be6ad82526d62a46 (table test-table [id=eeb8ce94f85e4a878ca24dfd177d9e83]) successfully deleted
I20260808 04:35:13.345736 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 26397
I20260808 04:35:13.348908 26884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:13.345890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:45007 (local address 127.0.0.1:52664)
0808 04:35:13.348708 (+ 2818us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:13.348785 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.65:45007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2702}
W20260808 04:35:13.349320 26129 catalog_manager.cc:4584] TS 0eba8ed56d6047daa77802d8f0ffc71c (127.15.228.65:45007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 526077d6c83f4b44be6ad82526d62a46: Network error: Client connection negotiation failed: client connection to 127.15.228.65:45007: connect: Connection refused (error 111)
I20260808 04:35:13.358530 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 26583
I20260808 04:35:13.371833 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 25697
I20260808 04:35:13.388375 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 25842
2026-08-08T04:35:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7915 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:35:13 dist-test-slave-drl0 krb5kdc[26924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:35:13 dist-test-slave-drl0 krb5kdc[26924](info): set up 2 sockets
Aug 08 04:35:13 dist-test-slave-drl0 krb5kdc[26924](info): commencing operation
krb5kdc: starting...
W20260808 04:35:14.690825 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.270s user 0.004s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:35:14 dist-test-slave-drl0 krb5kdc[26924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:35:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:35:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.126.keytab.
Entry for principal oryx/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.126.keytab.
I20260808 04:35:14.846230 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:44229
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.126.keytab
--principal=oryx/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:44361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:44229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:14.983883 26944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:14.984297 26944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:14.984347 26944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:14.990031 26944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:35:14.990150 26944 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 04:35:14.990185 26944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:14.990245 26944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:14.990286 26944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:35:14.990311 26944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:35:14.997534 26944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:44361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:44229
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.126.keytab
--principal=oryx/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:44229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:14.998795 26944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:14.999951 26944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:15.009030 26951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:15.009011 26954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:15.009052 26950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:15.009496 26944 server_base.cc:1034] running on GCE node
I20260808 04:35:15.010102 26944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:15.011318 26944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:15.012495 26944 hybrid_clock.cc:648] HybridClock initialized: now 1786163715012458 us; error 50 us; skew 500 ppm
Aug 08 04:35:15 dist-test-slave-drl0 krb5kdc[26924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163715, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:15.015411 26944 init.cc:377] Logged in from keytab as oryx/127.15.228.126@KRBTEST.COM (short username oryx)
I20260808 04:35:15.016386 26944 webserver.cc:460] Webserver started at http://127.15.228.126:42209/ using document root <none> and password file <none>
I20260808 04:35:15.016920 26944 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:15.016985 26944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:15.017150 26944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:15.019274 26944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "9c3fa82e61aa4dbbbd8f1d9561f487e6"
format_stamp: "Formatted at 2026-08-08 04:35:15 on dist-test-slave-drl0"
server_key: "2b3fea32c81f1dafb24a5394b4449159"
server_key_iv: "2b22f875be6ef291caed4e864880fb60"
server_key_version: "encryptionkey@0"
I20260808 04:35:15.019829 26944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "9c3fa82e61aa4dbbbd8f1d9561f487e6"
format_stamp: "Formatted at 2026-08-08 04:35:15 on dist-test-slave-drl0"
server_key: "2b3fea32c81f1dafb24a5394b4449159"
server_key_iv: "2b22f875be6ef291caed4e864880fb60"
server_key_version: "encryptionkey@0"
I20260808 04:35:15.024561 26944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 04:35:15.027704 26963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:15.029021 26944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:35:15.029127 26944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "9c3fa82e61aa4dbbbd8f1d9561f487e6"
format_stamp: "Formatted at 2026-08-08 04:35:15 on dist-test-slave-drl0"
server_key: "2b3fea32c81f1dafb24a5394b4449159"
server_key_iv: "2b22f875be6ef291caed4e864880fb60"
server_key_version: "encryptionkey@0"
I20260808 04:35:15.029213 26944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:15.047812 26944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:15.051268 26944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:15.051492 26944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:15.062428 26944 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:44229
I20260808 04:35:15.062543 27038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:44229 every 8 connection(s)
I20260808 04:35:15.063890 26944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:35:15.066386 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 26944
I20260808 04:35:15.066566 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:35:15.066906 16273 external_mini_cluster.cc:1428] Setting key 0115c018e2353785986079be9e6ebb73
I20260808 04:35:15.069232 27039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:15.076227 27039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6: Bootstrap starting.
Aug 08 04:35:15 dist-test-slave-drl0 krb5kdc[26924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM
I20260808 04:35:15.079101 27039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:15.080364 27039 log.cc:824] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:15.083554 27039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6: No bootstrap required, opened a new log
I20260808 04:35:15.085552 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:15.068901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46154 (local address 127.15.228.126:44229)
0808 04:35:15.069626 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:15.069646 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:15.069694 (+ 48us) server_negotiation.cc:407] Connection header received
0808 04:35:15.070450 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:15.070471 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:15.070878 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:15.071597 (+ 719us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:15.072728 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.073923 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.074976 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.075753 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.076000 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:15.079329 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:15.079344 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:15.079366 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:15.079414 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:15.082416 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:15.082991 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:15.083004 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:15.083011 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:15.083132 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:15.083445 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:15.083449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:15.083451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:15.083955 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:15.084160 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:15.084489 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 04:35:15.084816 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":247,"threads_started":1}
I20260808 04:35:15.087541 27039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3fa82e61aa4dbbbd8f1d9561f487e6" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 44229 } }
I20260808 04:35:15.087800 27039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:15.087888 27039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c3fa82e61aa4dbbbd8f1d9561f487e6, State: Initialized, Role: FOLLOWER
I20260808 04:35:15.088388 27039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3fa82e61aa4dbbbd8f1d9561f487e6" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 44229 } }
I20260808 04:35:15.088557 27039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:35:15.088646 27039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:35:15.088778 27039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:15.089753 27039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3fa82e61aa4dbbbd8f1d9561f487e6" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 44229 } }
I20260808 04:35:15.090102 27039 leader_election.cc:302] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c3fa82e61aa4dbbbd8f1d9561f487e6; no voters:
I20260808 04:35:15.090490 27039 leader_election.cc:288] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:35:15.090839 27044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:35:15.091279 27044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c3fa82e61aa4dbbbd8f1d9561f487e6, State: Running, Role: LEADER
I20260808 04:35:15.091805 27044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3fa82e61aa4dbbbd8f1d9561f487e6" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 44229 } }
I20260808 04:35:15.091845 27039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:35:15.093936 27045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c3fa82e61aa4dbbbd8f1d9561f487e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3fa82e61aa4dbbbd8f1d9561f487e6" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 44229 } } }
I20260808 04:35:15.093990 27046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c3fa82e61aa4dbbbd8f1d9561f487e6. Latest consensus state: current_term: 1 leader_uuid: "9c3fa82e61aa4dbbbd8f1d9561f487e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3fa82e61aa4dbbbd8f1d9561f487e6" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 44229 } } }
I20260808 04:35:15.094058 27045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:15.094089 27046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6 [sys.catalog]: This master's current role is: LEADER
W20260808 04:35:15.096947 27062 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:35:15.097019 27062 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:35:15.097106 27064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:35:15.098013 27064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:35:15.108695 27064 catalog_manager.cc:1347] Generated new cluster ID: ddb52c28c0694ce9b8639cab97306d4b
I20260808 04:35:15.108834 27064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:35:15.138504 27064 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:35:15.139920 27064 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:35:15.151441 27064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6: Generated new TSK 0
I20260808 04:35:15.152853 27064 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.65.keytab.
Entry for principal oryx/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.65.keytab.
I20260808 04:35:15.192201 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:44229
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.65.keytab
--principal=oryx/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:44361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:15.320235 27072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:15.320521 27072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:15.320572 27072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:15.325248 27072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:15.325430 27072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:15.325500 27072 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:35:15.325634 27072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:35:15.331245 27072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:44361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.65.keytab
--principal=oryx/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.228.126:44229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:15.332324 27072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:15.333487 27072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:15.343624 27086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:15.343717 27082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:15.343843 27072 server_base.cc:1034] running on GCE node
W20260808 04:35:15.343717 27079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:15.344285 27072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:15.344957 27072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:15.346342 27072 hybrid_clock.cc:648] HybridClock initialized: now 1786163715346270 us; error 77 us; skew 500 ppm
Aug 08 04:35:15 dist-test-slave-drl0 krb5kdc[26924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163715, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:15.349936 27072 init.cc:377] Logged in from keytab as oryx/127.15.228.65@KRBTEST.COM (short username oryx)
I20260808 04:35:15.351011 27072 webserver.cc:460] Webserver started at http://127.15.228.65:46645/ using document root <none> and password file <none>
I20260808 04:35:15.351588 27072 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:15.351655 27072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:15.351828 27072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:15.354082 27072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "128cdb84c72d4f34a433ff04613d886d"
format_stamp: "Formatted at 2026-08-08 04:35:15 on dist-test-slave-drl0"
server_key: "59e02c921bc9aeeeab3a52d06c845e37"
server_key_iv: "b7e02ba713098a6d6c1d4c486129dc65"
server_key_version: "encryptionkey@0"
I20260808 04:35:15.354652 27072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "128cdb84c72d4f34a433ff04613d886d"
format_stamp: "Formatted at 2026-08-08 04:35:15 on dist-test-slave-drl0"
server_key: "59e02c921bc9aeeeab3a52d06c845e37"
server_key_iv: "b7e02ba713098a6d6c1d4c486129dc65"
server_key_version: "encryptionkey@0"
I20260808 04:35:15.359128 27072 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260808 04:35:15.361961 27098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:15.363358 27072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:35:15.363502 27072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "128cdb84c72d4f34a433ff04613d886d"
format_stamp: "Formatted at 2026-08-08 04:35:15 on dist-test-slave-drl0"
server_key: "59e02c921bc9aeeeab3a52d06c845e37"
server_key_iv: "b7e02ba713098a6d6c1d4c486129dc65"
server_key_version: "encryptionkey@0"
I20260808 04:35:15.363619 27072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:15.409657 27072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:15.413317 27072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:15.413558 27072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:15.415390 27072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:15.415457 27072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:15.415529 27072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:15.415581 27072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:35:15 dist-test-slave-drl0 krb5kdc[26924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163715, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM
I20260808 04:35:15.427510 27072 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:38997
I20260808 04:35:15.427709 27246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:38997 every 8 connection(s)
I20260808 04:35:15.428623 27072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:35:15.430807 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:15.416274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:60977 (local address 127.15.228.126:44229)
0808 04:35:15.416469 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:15.416476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:15.417460 (+ 984us) server_negotiation.cc:407] Connection header received
0808 04:35:15.418742 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:15.418745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:15.418802 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:15.418910 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:15.420719 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.421417 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.423354 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.423752 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.423895 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:15.426988 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:15.426992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:15.426994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:15.427020 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:15.428660 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:15.429249 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:15.429252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:15.429255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:15.429346 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:15.429712 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:15.429715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:15.429716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:15.429916 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:15.430003 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:15.430539 (+ 536us) server_negotiation.cc:299] Negotiation successful
0808 04:35:15.430661 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:35:15.431578 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 27072
I20260808 04:35:15.431622 27109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:15.416684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:44229 (local address 127.15.228.65:60977)
0808 04:35:15.417236 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:15.417299 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:15.418476 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:15.419145 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:15.419170 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:15.419722 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:15.420482 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:15.420506 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.421566 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.421571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:15.423203 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:15.423218 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.423971 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.423976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:15.424157 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:15.424831 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:15.424849 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:15.426658 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:15.428805 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:15.428811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:15.428825 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:15.429130 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:15.429447 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:15.429451 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:15.429453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:15.429618 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:15.430041 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:15.430046 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:15.430275 (+ 229us) client_negotiation.cc:769] Sending connection context
0808 04:35:15.430445 (+ 170us) client_negotiation.cc:240] Negotiation successful
0808 04:35:15.430692 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":157,"threads_started":1}
I20260808 04:35:15.431751 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:35:15.432093 16273 external_mini_cluster.cc:1428] Setting key 73ca06b831e384c4811078fa46ae741d
I20260808 04:35:15.433019 27248 heartbeater.cc:347] Connected to a master server at 127.15.228.126:44229
I20260808 04:35:15.433184 27248 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:15.433689 27248 heartbeater.cc:511] Master 127.15.228.126:44229 requested a full tablet report, sending...
I20260808 04:35:15.436106 26985 ts_manager.cc:194] Registered new tserver with Master: 128cdb84c72d4f34a433ff04613d886d (127.15.228.65:38997)
I20260808 04:35:15.437589 26985 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:60977
WARNING: no policy specified for oryx/127.15.228.66@KRBTEST.COM; defaulting to no policy
I20260808 04:35:15.447997 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:15.438671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:45307 (local address 127.15.228.126:44229)
0808 04:35:15.439033 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:15.439038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:15.439056 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:35:15.439093 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:15.439095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:15.439138 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:15.439232 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:15.439885 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.440699 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.442016 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.442240 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.442386 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:15.444301 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:15.444305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:15.444308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:15.444340 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:15.446101 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:15.446684 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:15.446687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:15.446689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:15.446780 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:15.447126 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:15.447130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:15.447132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:15.447322 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:15.447441 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:15.447717 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 04:35:15.447831 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260808 04:35:15.448014 27257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:15.435508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:44229 (local address 127.15.228.65:45307)
0808 04:35:15.435826 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:15.435842 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:15.435946 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:15.439276 (+ 3330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:15.439281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:15.439546 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:15.439734 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:15.439744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.440853 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.440859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:15.441851 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:15.441867 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.442394 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.442397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:15.442499 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:15.443331 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:15.443358 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:15.444005 (+ 647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:15.446281 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:15.446287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:15.446292 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:15.446571 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:15.446883 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:15.446886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:15.446888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:15.447027 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:15.447429 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:15.447432 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:15.447537 (+ 105us) client_negotiation.cc:769] Sending connection context
0808 04:35:15.447647 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 04:35:15.447817 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.228.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.66.keytab.
Entry for principal oryx/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.66.keytab.
I20260808 04:35:15.470679 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:44229
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.66.keytab
--principal=oryx/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:44361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:15.596937 27263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:15.597216 27263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:15.597265 27263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:15.601610 27263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:15.601687 27263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:15.601734 27263 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:35:15.601773 27263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:35:15.607100 27263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:44361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.66.keytab
--principal=oryx/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.228.126:44229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:15.608206 27263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:15.609339 27263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:15.619365 27276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:15.619385 27273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:15.619351 27272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:15.619791 27263 server_base.cc:1034] running on GCE node
I20260808 04:35:15.620433 27263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:15.621104 27263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:15.622345 27263 hybrid_clock.cc:648] HybridClock initialized: now 1786163715622296 us; error 61 us; skew 500 ppm
Aug 08 04:35:15 dist-test-slave-drl0 krb5kdc[26924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163715, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:15.625566 27263 init.cc:377] Logged in from keytab as oryx/127.15.228.66@KRBTEST.COM (short username oryx)
I20260808 04:35:15.626469 27263 webserver.cc:460] Webserver started at http://127.15.228.66:33459/ using document root <none> and password file <none>
I20260808 04:35:15.627239 27263 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:15.627341 27263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:15.627576 27263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:15.629807 27263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc"
format_stamp: "Formatted at 2026-08-08 04:35:15 on dist-test-slave-drl0"
server_key: "4666fd5c712a9949a545ebc255bd9574"
server_key_iv: "5968863bb57174ece34089c6b13dae5d"
server_key_version: "encryptionkey@0"
I20260808 04:35:15.630409 27263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc"
format_stamp: "Formatted at 2026-08-08 04:35:15 on dist-test-slave-drl0"
server_key: "4666fd5c712a9949a545ebc255bd9574"
server_key_iv: "5968863bb57174ece34089c6b13dae5d"
server_key_version: "encryptionkey@0"
I20260808 04:35:15.634584 27263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260808 04:35:15.637769 27286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:15.639492 27263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:35:15.639649 27263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc"
format_stamp: "Formatted at 2026-08-08 04:35:15 on dist-test-slave-drl0"
server_key: "4666fd5c712a9949a545ebc255bd9574"
server_key_iv: "5968863bb57174ece34089c6b13dae5d"
server_key_version: "encryptionkey@0"
I20260808 04:35:15.639771 27263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:15.670585 27263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:15.674048 27263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:15.674269 27263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:15.675997 27263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:15.676075 27263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:15.676151 27263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:15.676195 27263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:35:15 dist-test-slave-drl0 krb5kdc[26924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163715, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM
I20260808 04:35:15.687927 27263 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:45041
I20260808 04:35:15.688027 27430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:45041 every 8 connection(s)
I20260808 04:35:15.689567 27263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:35:15.690328 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:15.677310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:59169 (local address 127.15.228.126:44229)
0808 04:35:15.677490 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:15.677498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:15.678338 (+ 840us) server_negotiation.cc:407] Connection header received
0808 04:35:15.679806 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:15.679810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:15.679868 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:15.679954 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:15.681194 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.681735 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.682858 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.683156 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.683274 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:15.686476 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:15.686480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:15.686483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:15.686513 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:15.688192 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:15.688716 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:15.688720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:15.688722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:15.688808 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:15.689179 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:15.689184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:15.689185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:15.689381 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:15.689493 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:15.690029 (+ 536us) server_negotiation.cc:299] Negotiation successful
0808 04:35:15.690154 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 04:35:15.691318 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:15.677640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:44229 (local address 127.15.228.66:59169)
0808 04:35:15.678149 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:15.678204 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:15.679511 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:15.680140 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:15.680146 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:15.680510 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:15.681020 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:15.681035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.681871 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.681875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:15.682725 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:15.682734 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.683330 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.683334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:15.683554 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:15.684241 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:15.684263 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:15.686296 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:15.688318 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:15.688324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:15.688338 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:15.688603 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:15.688913 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:15.688917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:15.688919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:15.689081 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:15.689493 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:15.689500 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:15.689784 (+ 284us) client_negotiation.cc:769] Sending connection context
0808 04:35:15.689983 (+ 199us) client_negotiation.cc:240] Negotiation successful
0808 04:35:15.690296 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":117,"threads_started":1}
I20260808 04:35:15.692873 27431 heartbeater.cc:347] Connected to a master server at 127.15.228.126:44229
I20260808 04:35:15.693135 27431 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:15.693969 27431 heartbeater.cc:511] Master 127.15.228.126:44229 requested a full tablet report, sending...
I20260808 04:35:15.695541 26988 ts_manager.cc:194] Registered new tserver with Master: 0cf16ea67b6a4b66956c5dc43c0ca4fc (127.15.228.66:45041)
I20260808 04:35:15.696367 26988 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:59169
I20260808 04:35:15.699659 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 27263
I20260808 04:35:15.699800 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:35:15.700088 16273 external_mini_cluster.cc:1428] Setting key 6c4cd7765b00b3638f6fc1e87f97bf5e
I20260808 04:35:15.704741 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:15.695566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:59811 (local address 127.15.228.126:44229)
0808 04:35:15.695772 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:15.695776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:15.695954 (+ 178us) server_negotiation.cc:407] Connection header received
0808 04:35:15.696074 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:15.696077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:15.696121 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:15.696213 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:15.696831 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.697585 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.698666 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.698917 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.699142 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:15.700702 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:15.700705 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:15.700708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:15.700776 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:15.702918 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:15.703465 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:15.703469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:15.703471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:15.703555 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:15.703840 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:15.703843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:15.703845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:15.704066 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:15.704180 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:15.704460 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 04:35:15.704579 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260808 04:35:15.704759 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:15.695471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:44229 (local address 127.15.228.66:59811)
0808 04:35:15.695841 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:15.695859 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:15.695995 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:15.696296 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:15.696300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:15.696546 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:15.696720 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:15.696729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.697799 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.697804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:15.698500 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:15.698510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.699163 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.699168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:15.699281 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:15.699969 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:15.699985 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:15.700502 (+ 517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:15.703114 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:15.703119 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:15.703121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:15.703357 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:15.703652 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:15.703656 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:15.703658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:15.703745 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:15.704170 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:15.704173 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:15.704281 (+ 108us) client_negotiation.cc:769] Sending connection context
0808 04:35:15.704391 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 04:35:15.704566 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":154,"threads_started":1}
WARNING: no policy specified for oryx/127.15.228.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.228.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.67.keytab.
Entry for principal oryx/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.67.keytab.
I20260808 04:35:15.737749 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:44229
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.67.keytab
--principal=oryx/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:44361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:15.862251 27443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:15.862512 27443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:15.862556 27443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:15.866580 27443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:15.866640 27443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:15.866797 27443 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 04:35:15.866884 27443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:35:15.873775 27443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:44361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.67.keytab
--principal=oryx/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.228.126:44229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:15.874871 27443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:15.875933 27443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:15.887029 27451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:15.887117 27455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:15.887030 27449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:15.887331 27443 server_base.cc:1034] running on GCE node
I20260808 04:35:15.887890 27443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:15.888526 27443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:15.889732 27443 hybrid_clock.cc:648] HybridClock initialized: now 1786163715889654 us; error 103 us; skew 500 ppm
Aug 08 04:35:15 dist-test-slave-drl0 krb5kdc[26924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163715, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:15.893501 27443 init.cc:377] Logged in from keytab as oryx/127.15.228.67@KRBTEST.COM (short username oryx)
I20260808 04:35:15.894495 27443 webserver.cc:460] Webserver started at http://127.15.228.67:46599/ using document root <none> and password file <none>
I20260808 04:35:15.895159 27443 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:15.895232 27443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:15.895449 27443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:15.897490 27443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "63e28fde95f3406499409d4fd6ac31ea"
format_stamp: "Formatted at 2026-08-08 04:35:15 on dist-test-slave-drl0"
server_key: "68baf5752f573147ef1c9083de27649e"
server_key_iv: "d723c5abd483fa9323917f0123c89764"
server_key_version: "encryptionkey@0"
I20260808 04:35:15.898062 27443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "63e28fde95f3406499409d4fd6ac31ea"
format_stamp: "Formatted at 2026-08-08 04:35:15 on dist-test-slave-drl0"
server_key: "68baf5752f573147ef1c9083de27649e"
server_key_iv: "d723c5abd483fa9323917f0123c89764"
server_key_version: "encryptionkey@0"
I20260808 04:35:15.902040 27443 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:35:15.905050 27463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:15.906563 27443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:35:15.906721 27443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "63e28fde95f3406499409d4fd6ac31ea"
format_stamp: "Formatted at 2026-08-08 04:35:15 on dist-test-slave-drl0"
server_key: "68baf5752f573147ef1c9083de27649e"
server_key_iv: "d723c5abd483fa9323917f0123c89764"
server_key_version: "encryptionkey@0"
I20260808 04:35:15.906843 27443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:15.936779 27443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:15.940781 27443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:15.941025 27443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:15.942893 27443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:15.943012 27443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:15.943110 27443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:15.943164 27443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 04:35:15 dist-test-slave-drl0 krb5kdc[26924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163715, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM
I20260808 04:35:15.957199 27443 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:37655
I20260808 04:35:15.957180 27621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:37655 every 8 connection(s)
I20260808 04:35:15.958679 27443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:35:15.961602 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:15.945030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:59439 (local address 127.15.228.126:44229)
0808 04:35:15.945252 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:15.945258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:15.947063 (+ 1805us) server_negotiation.cc:407] Connection header received
0808 04:35:15.948710 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:15.948714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:15.948797 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:15.948895 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:15.950407 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.951465 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.953165 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.953411 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.953527 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:15.957232 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:15.957237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:15.957240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:15.957276 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:15.959415 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:15.960016 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:15.960021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:15.960023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:15.960111 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:15.960474 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:15.960479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:15.960481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:15.960681 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:15.960794 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:15.961302 (+ 508us) server_negotiation.cc:299] Negotiation successful
0808 04:35:15.961433 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 04:35:15.962548 27486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:15.945772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:44229 (local address 127.15.228.67:59439)
0808 04:35:15.946780 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:15.946846 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:15.948323 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:15.949247 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:15.949257 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:15.949646 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:15.950180 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:15.950196 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.951640 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.951644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:15.953022 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:15.953033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.953592 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.953597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:15.953786 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:15.954691 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:15.954717 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:15.956821 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:15.959575 (+ 2754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:15.959584 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:15.959605 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:15.959892 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:15.960213 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:15.960217 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:15.960219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:15.960370 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:15.960791 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:15.960799 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:15.961063 (+ 264us) client_negotiation.cc:769] Sending connection context
0808 04:35:15.961261 (+ 198us) client_negotiation.cc:240] Negotiation successful
0808 04:35:15.961598 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":315,"threads_started":1}
I20260808 04:35:15.964387 27622 heartbeater.cc:347] Connected to a master server at 127.15.228.126:44229
I20260808 04:35:15.964586 27622 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:15.965142 27622 heartbeater.cc:511] Master 127.15.228.126:44229 requested a full tablet report, sending...
I20260808 04:35:15.966289 26988 ts_manager.cc:194] Registered new tserver with Master: 63e28fde95f3406499409d4fd6ac31ea (127.15.228.67:37655)
I20260808 04:35:15.967036 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 27443
I20260808 04:35:15.967197 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:35:15.967214 26988 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:59439
I20260808 04:35:15.967667 16273 external_mini_cluster.cc:1428] Setting key 4290df5f057d1b6dc536baa9f40d4eb4
I20260808 04:35:15.969727 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:35:15.977380 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:15.966862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:54213 (local address 127.15.228.126:44229)
0808 04:35:15.967024 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:15.967030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:15.967172 (+ 142us) server_negotiation.cc:407] Connection header received
0808 04:35:15.967365 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:15.967368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:15.967412 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:15.967465 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:15.968107 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.968854 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.970516 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.971187 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.971326 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:15.973006 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:15.973011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:15.973016 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:15.973048 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:15.974886 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:15.975830 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:15.975836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:15.975838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:15.975941 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:15.976263 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:15.976268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:15.976270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:15.976503 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:15.976626 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:15.977075 (+ 449us) server_negotiation.cc:299] Negotiation successful
0808 04:35:15.977207 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:35:15.977401 27629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:15.966741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:44229 (local address 127.15.228.67:54213)
0808 04:35:15.967094 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:15.967108 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:15.967242 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:15.967598 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:15.967601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:15.967820 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:15.967939 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:15.967944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.969511 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.969515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:15.970373 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:15.970385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.971437 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.971447 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:15.971735 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:15.972324 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:15.972345 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:15.972835 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:15.975342 (+ 2507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:15.975347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:15.975350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:15.975664 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:15.976049 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:15.976053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:15.976055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:15.976157 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:15.976610 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:15.976613 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:15.976824 (+ 211us) client_negotiation.cc:769] Sending connection context
0808 04:35:15.977001 (+ 177us) client_negotiation.cc:240] Negotiation successful
0808 04:35:15.977184 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":107,"threads_started":1}
Aug 08 04:35:15 dist-test-slave-drl0 krb5kdc[26924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163715, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 04:35:15 dist-test-slave-drl0 krb5kdc[26924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:15 dist-test-slave-drl0 krb5kdc[26924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 04:35:16.001655 27042 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:15.994737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46168 (local address 127.15.228.126:44229)
0808 04:35:15.995041 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:15.995046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:15.995073 (+ 27us) server_negotiation.cc:407] Connection header received
0808 04:35:15.995119 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:15.995121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:15.995168 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:15.995256 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:15.996090 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.996630 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.997537 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:15.997747 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:15.997841 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.001524 (+ 3683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46168: BlockingRecv error: recv got EOF from 127.0.0.1:46168 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Aug 08 04:35:16 dist-test-slave-drl0 krb5kdc[26924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163715, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM
I20260808 04:35:16.019935 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.003907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46176 (local address 127.15.228.126:44229)
0808 04:35:16.004147 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:16.004152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:16.004407 (+ 255us) server_negotiation.cc:407] Connection header received
0808 04:35:16.004516 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:16.004518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:16.004562 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:16.004647 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:16.005584 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.006070 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.006987 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.007221 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.007318 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.010965 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:16.010973 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:16.010978 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:16.011013 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:16.017785 (+ 6772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:16.018382 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:16.018388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:16.018390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:16.018578 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:16.018959 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:16.018964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:16.018969 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:16.019194 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:16.019352 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:16.019600 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 04:35:16.019760 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260808 04:35:16.024475 26988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:35:16.028975 26988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:35:16.049757 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.042795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:37655 (local address 127.0.0.1:49568)
0808 04:35:16.044135 (+ 1340us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:16.044149 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:16.044291 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:16.045094 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:16.045097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:16.045117 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:16.045214 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.045220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.046456 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.046460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.047661 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.047672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.049134 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.049137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.049293 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.049349 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 04:35:16.049465 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 04:35:16.049586 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1235,"mutex_wait_us":1023,"thread_start_us":135,"threads_started":1}
I20260808 04:35:16.049993 27651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.043102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49568 (local address 127.15.228.67:37655)
0808 04:35:16.043692 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:16.043709 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:16.044222 (+ 513us) server_negotiation.cc:407] Connection header received
0808 04:35:16.044435 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:16.044440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:16.044590 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:16.044726 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:16.045345 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.046319 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.047824 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.048698 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.048789 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.049018 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:16.049611 (+ 593us) server_negotiation.cc:299] Negotiation successful
0808 04:35:16.049792 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":149,"threads_started":1}
I20260808 04:35:16.049983 27649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.042795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:45041 (local address 127.0.0.1:45368)
0808 04:35:16.043495 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:16.043552 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:16.043687 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:16.044704 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:16.044710 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:16.044735 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:16.044880 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.044887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.046319 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.046323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.047899 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.047911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.049078 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.049083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.049274 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.049331 (+ 57us) client_negotiation.cc:769] Sending connection context
0808 04:35:16.049579 (+ 248us) client_negotiation.cc:240] Negotiation successful
0808 04:35:16.049706 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":73,"thread_start_us":101,"threads_started":1}
I20260808 04:35:16.050416 27653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.043551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.15.228.66:45041)
0808 04:35:16.044172 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:16.044185 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:16.044208 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:35:16.044288 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:16.044293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:16.044531 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:16.044701 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:16.044988 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.046167 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.048013 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.048807 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.048915 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.049069 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:16.050107 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0808 04:35:16.050261 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":111,"threads_started":1}
I20260808 04:35:16.051026 27655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.043086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:38997 (local address 127.0.0.1:47182)
0808 04:35:16.044019 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:16.044032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:16.044121 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:16.045188 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:16.045190 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:16.045206 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:16.045315 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.045323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.046585 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.046589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.047899 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.047911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.050257 (+ 2346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.050262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.050673 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.050699 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 04:35:16.050771 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 04:35:16.050858 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":71,"thread_start_us":140,"threads_started":1}
I20260808 04:35:16.051553 27652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.043509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47182 (local address 127.15.228.65:38997)
0808 04:35:16.044604 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:16.044609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:16.044650 (+ 41us) server_negotiation.cc:407] Connection header received
0808 04:35:16.044722 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:16.044727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:16.044894 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:16.045042 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:16.045426 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.046472 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.049132 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.049945 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.050051 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.050184 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:16.051226 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0808 04:35:16.051341 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":943,"thread_start_us":133,"threads_started":1}
I20260808 04:35:16.054185 27343 tablet_service.cc:1467] Processing CreateTablet for tablet ed635a10182847cda2c6ad79cdeeef18 (DEFAULT_TABLE table=test-table [id=75db03d57a73406895e2b3ba6cc065f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:16.054276 27534 tablet_service.cc:1467] Processing CreateTablet for tablet ed635a10182847cda2c6ad79cdeeef18 (DEFAULT_TABLE table=test-table [id=75db03d57a73406895e2b3ba6cc065f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:16.055733 27343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed635a10182847cda2c6ad79cdeeef18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:16.055787 27534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed635a10182847cda2c6ad79cdeeef18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:16.058606 27164 tablet_service.cc:1467] Processing CreateTablet for tablet ed635a10182847cda2c6ad79cdeeef18 (DEFAULT_TABLE table=test-table [id=75db03d57a73406895e2b3ba6cc065f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:16.059778 27164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed635a10182847cda2c6ad79cdeeef18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:16.065847 27658 tablet_bootstrap.cc:492] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea: Bootstrap starting.
I20260808 04:35:16.066923 27662 tablet_bootstrap.cc:492] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d: Bootstrap starting.
I20260808 04:35:16.067803 27660 tablet_bootstrap.cc:492] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Bootstrap starting.
I20260808 04:35:16.068553 27658 tablet_bootstrap.cc:654] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:16.069334 27662 tablet_bootstrap.cc:654] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:16.069561 27658 log.cc:824] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:16.069998 27660 tablet_bootstrap.cc:654] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:16.070320 27662 log.cc:824] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:16.070811 27660 log.cc:824] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:16.071900 27658 tablet_bootstrap.cc:492] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea: No bootstrap required, opened a new log
I20260808 04:35:16.072103 27658 ts_tablet_manager.cc:1397] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:35:16.072171 27662 tablet_bootstrap.cc:492] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d: No bootstrap required, opened a new log
I20260808 04:35:16.072438 27662 ts_tablet_manager.cc:1397] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:35:16.072727 27660 tablet_bootstrap.cc:492] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: No bootstrap required, opened a new log
I20260808 04:35:16.072892 27660 ts_tablet_manager.cc:1397] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:35:16.076913 27662 raft_consensus.cc:348] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } } peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } }
I20260808 04:35:16.077188 27662 raft_consensus.cc:374] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:16.077250 27662 raft_consensus.cc:729] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128cdb84c72d4f34a433ff04613d886d, State: Initialized, Role: FOLLOWER
I20260808 04:35:16.077137 27658 raft_consensus.cc:348] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } } peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } }
I20260808 04:35:16.077543 27658 raft_consensus.cc:374] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:16.077622 27658 raft_consensus.cc:729] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e28fde95f3406499409d4fd6ac31ea, State: Initialized, Role: FOLLOWER
I20260808 04:35:16.077489 27660 raft_consensus.cc:348] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } } peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } }
I20260808 04:35:16.077786 27660 raft_consensus.cc:374] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:16.077845 27660 raft_consensus.cc:729] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf16ea67b6a4b66956c5dc43c0ca4fc, State: Initialized, Role: FOLLOWER
I20260808 04:35:16.078137 27662 consensus_queue.cc:260] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } } peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } }
I20260808 04:35:16.078339 27658 consensus_queue.cc:260] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } } peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } }
I20260808 04:35:16.078544 27660 consensus_queue.cc:260] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } } peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } }
I20260808 04:35:16.079382 27662 ts_tablet_manager.cc:1428] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20260808 04:35:16.079643 27658 ts_tablet_manager.cc:1428] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260808 04:35:16.079643 27660 ts_tablet_manager.cc:1428] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 04:35:16.079799 27248 heartbeater.cc:503] Master 127.15.228.126:44229 was elected leader, sending a full tablet report...
I20260808 04:35:16.079989 27622 heartbeater.cc:503] Master 127.15.228.126:44229 was elected leader, sending a full tablet report...
I20260808 04:35:16.080878 27431 heartbeater.cc:503] Master 127.15.228.126:44229 was elected leader, sending a full tablet report...
I20260808 04:35:16.098982 27670 raft_consensus.cc:482] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:35:16.099385 27670 raft_consensus.cc:504] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } } peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } }
I20260808 04:35:16.101148 27670 leader_election.cc:288] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63e28fde95f3406499409d4fd6ac31ea (127.15.228.67:37655), 128cdb84c72d4f34a433ff04613d886d (127.15.228.65:38997)
I20260808 04:35:16.106292 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.101585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:38997 (local address 127.15.228.66:44151)
0808 04:35:16.101866 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:16.101887 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:16.102028 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:16.102357 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:16.102360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:16.102380 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:16.102539 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.102546 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.103591 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.103596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.104850 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.104863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.105742 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.105749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.105890 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.105941 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 04:35:16.106044 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 04:35:16.106145 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260808 04:35:16.106451 27652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.101782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:44151 (local address 127.15.228.65:38997)
0808 04:35:16.102040 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:16.102046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:16.102067 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:35:16.102168 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:16.102172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:16.102231 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:16.102322 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:16.102698 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.103445 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.105022 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.105508 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.105645 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.105704 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:16.106137 (+ 433us) server_negotiation.cc:299] Negotiation successful
0808 04:35:16.106297 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260808 04:35:16.107301 27191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed635a10182847cda2c6ad79cdeeef18" candidate_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128cdb84c72d4f34a433ff04613d886d" is_pre_election: true
I20260808 04:35:16.107481 27671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.101585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:37655 (local address 127.15.228.66:49485)
0808 04:35:16.102584 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:16.102600 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:16.102705 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:16.103061 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:16.103064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:16.103082 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:16.103719 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.103728 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.104594 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.104598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.105812 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.105833 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.107028 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.107032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.107134 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.107158 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:35:16.107250 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 04:35:16.107328 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"mutex_wait_us":444,"thread_start_us":224,"threads_started":1}
I20260808 04:35:16.107558 27651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.101862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:49485 (local address 127.15.228.67:37655)
0808 04:35:16.102013 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:16.102019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:16.102689 (+ 670us) server_negotiation.cc:407] Connection header received
0808 04:35:16.102842 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:16.102844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:16.102896 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:16.103053 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:16.103886 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.104420 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.106129 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.106810 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.107046 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.107092 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:16.107308 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 04:35:16.107389 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:35:16.107721 27191 raft_consensus.cc:2454] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cf16ea67b6a4b66956c5dc43c0ca4fc in term 0.
I20260808 04:35:16.108330 27287 leader_election.cc:302] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf16ea67b6a4b66956c5dc43c0ca4fc, 128cdb84c72d4f34a433ff04613d886d; no voters:
I20260808 04:35:16.108407 27561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed635a10182847cda2c6ad79cdeeef18" candidate_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63e28fde95f3406499409d4fd6ac31ea" is_pre_election: true
I20260808 04:35:16.108677 27561 raft_consensus.cc:2454] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cf16ea67b6a4b66956c5dc43c0ca4fc in term 0.
I20260808 04:35:16.108693 27670 raft_consensus.cc:2781] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:35:16.108800 27670 raft_consensus.cc:482] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:35:16.108861 27670 raft_consensus.cc:3037] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:16.110107 27670 raft_consensus.cc:504] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } } peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } }
I20260808 04:35:16.110648 27670 leader_election.cc:288] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [CANDIDATE]: Term 1 election: Requested vote from peers 63e28fde95f3406499409d4fd6ac31ea (127.15.228.67:37655), 128cdb84c72d4f34a433ff04613d886d (127.15.228.65:38997)
I20260808 04:35:16.111100 27561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed635a10182847cda2c6ad79cdeeef18" candidate_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63e28fde95f3406499409d4fd6ac31ea"
I20260808 04:35:16.111236 27561 raft_consensus.cc:3037] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:16.111536 27191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed635a10182847cda2c6ad79cdeeef18" candidate_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128cdb84c72d4f34a433ff04613d886d"
I20260808 04:35:16.111994 27191 raft_consensus.cc:3037] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:16.112967 27561 raft_consensus.cc:2454] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cf16ea67b6a4b66956c5dc43c0ca4fc in term 1.
I20260808 04:35:16.113413 27292 leader_election.cc:302] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf16ea67b6a4b66956c5dc43c0ca4fc, 63e28fde95f3406499409d4fd6ac31ea; no voters:
I20260808 04:35:16.113727 27670 raft_consensus.cc:2781] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:35:16.114065 27670 raft_consensus.cc:686] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 1 LEADER]: Becoming Leader. State: Replica: 0cf16ea67b6a4b66956c5dc43c0ca4fc, State: Running, Role: LEADER
I20260808 04:35:16.114318 27191 raft_consensus.cc:2454] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cf16ea67b6a4b66956c5dc43c0ca4fc in term 1.
I20260808 04:35:16.114622 27670 consensus_queue.cc:237] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } } peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } }
I20260808 04:35:16.119762 26985 catalog_manager.cc:5495] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cf16ea67b6a4b66956c5dc43c0ca4fc (127.15.228.66). New cstate: current_term: 1 leader_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } health_report { overall_health: UNKNOWN } } }
I20260808 04:35:16.142783 27653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.139006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45384 (local address 127.15.228.66:45041)
0808 04:35:16.139181 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:16.139186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:16.139202 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:35:16.139322 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:16.139324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:16.139382 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:16.139507 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:35:16.139777 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.140316 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.141226 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.141470 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.141588 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.141890 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:35:16.142361 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:35:16.142502 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:16.142563 (+ 61us) server_negotiation.cc:299] Negotiation successful
0808 04:35:16.142625 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 04:35:16.149959 27561 raft_consensus.cc:1261] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea [term 1 FOLLOWER]: Refusing update from remote peer 0cf16ea67b6a4b66956c5dc43c0ca4fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:35:16.149971 27191 raft_consensus.cc:1261] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d [term 1 FOLLOWER]: Refusing update from remote peer 0cf16ea67b6a4b66956c5dc43c0ca4fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:35:16.151283 27670 consensus_queue.cc:1035] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:35:16.151841 27672 consensus_queue.cc:1035] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:35:16.162703 27677 mvcc.cc:205] Tried to move back new op lower bound from 7316126581342695424 to 7316126581210378240. Current Snapshot: MvccSnapshot[applied={T|T < 7316126581342695424}]
I20260808 04:35:16.162907 27678 mvcc.cc:205] Tried to move back new op lower bound from 7316126581342695424 to 7316126581210378240. Current Snapshot: MvccSnapshot[applied={T|T < 7316126581342695424}]
I20260808 04:35:16.172525 26985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:35:16.172719 26985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:35:16.175709 26985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6: Sending DeleteTablet for 3 replicas of tablet ed635a10182847cda2c6ad79cdeeef18
I20260808 04:35:16.176647 27343 tablet_service.cc:1514] Processing DeleteTablet for tablet ed635a10182847cda2c6ad79cdeeef18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:16 UTC) from {username='oryx', principal='oryx/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:45368
I20260808 04:35:16.176645 27534 tablet_service.cc:1514] Processing DeleteTablet for tablet ed635a10182847cda2c6ad79cdeeef18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:16 UTC) from {username='oryx', principal='oryx/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:49568
I20260808 04:35:16.177260 27688 tablet_replica.cc:330] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: stopping tablet replica
I20260808 04:35:16.177268 27164 tablet_service.cc:1514] Processing DeleteTablet for tablet ed635a10182847cda2c6ad79cdeeef18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:16 UTC) from {username='oryx', principal='oryx/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:47182
I20260808 04:35:16.177626 27688 raft_consensus.cc:2229] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:35:16.177860 27690 tablet_replica.cc:330] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d: stopping tablet replica
I20260808 04:35:16.178099 27688 raft_consensus.cc:2258] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:16.178292 27690 raft_consensus.cc:2229] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:35:16.178694 27690 raft_consensus.cc:2258] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:16.179046 26983 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:35:16.179553 26983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:35:16.180063 27688 ts_tablet_manager.cc:1905] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:16.183017 27690 ts_tablet_manager.cc:1905] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:16.183069 27688 ts_tablet_manager.cc:1918] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:35:16.183168 27688 log.cc:1196] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/ed635a10182847cda2c6ad79cdeeef18
I20260808 04:35:16.183525 27688 ts_tablet_manager.cc:1939] T ed635a10182847cda2c6ad79cdeeef18 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Deleting consensus metadata
I20260808 04:35:16.184803 26966 catalog_manager.cc:4841] TS 0cf16ea67b6a4b66956c5dc43c0ca4fc (127.15.228.66:45041): tablet ed635a10182847cda2c6ad79cdeeef18 (table test-table [id=75db03d57a73406895e2b3ba6cc065f3]) successfully deleted
I20260808 04:35:16.184939 27689 tablet_replica.cc:330] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea: stopping tablet replica
I20260808 04:35:16.185259 27689 raft_consensus.cc:2229] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:35:16.185590 27689 raft_consensus.cc:2258] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:16.187125 27689 ts_tablet_manager.cc:1905] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:16.187644 27164 tablet_service.cc:1467] Processing CreateTablet for tablet cd6d2314599b4a87b994b9d99ace18d5 (DEFAULT_TABLE table=test-table [id=e8403ee1fdb14b488f38b61a88534112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:16.187830 27534 tablet_service.cc:1467] Processing CreateTablet for tablet cd6d2314599b4a87b994b9d99ace18d5 (DEFAULT_TABLE table=test-table [id=e8403ee1fdb14b488f38b61a88534112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:16.187945 27164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd6d2314599b4a87b994b9d99ace18d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:16.188086 27534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd6d2314599b4a87b994b9d99ace18d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:16.189769 27689 ts_tablet_manager.cc:1918] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:35:16.189854 27689 log.cc:1196] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/ed635a10182847cda2c6ad79cdeeef18
I20260808 04:35:16.190172 27689 ts_tablet_manager.cc:1939] T ed635a10182847cda2c6ad79cdeeef18 P 63e28fde95f3406499409d4fd6ac31ea: Deleting consensus metadata
I20260808 04:35:16.190505 27662 tablet_bootstrap.cc:492] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d: Bootstrap starting.
I20260808 04:35:16.190599 27658 tablet_bootstrap.cc:492] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea: Bootstrap starting.
I20260808 04:35:16.191490 26967 catalog_manager.cc:4841] TS 63e28fde95f3406499409d4fd6ac31ea (127.15.228.67:37655): tablet ed635a10182847cda2c6ad79cdeeef18 (table test-table [id=75db03d57a73406895e2b3ba6cc065f3]) successfully deleted
I20260808 04:35:16.191604 27658 tablet_bootstrap.cc:654] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:16.191689 27662 tablet_bootstrap.cc:654] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:16.192629 27343 tablet_service.cc:1467] Processing CreateTablet for tablet cd6d2314599b4a87b994b9d99ace18d5 (DEFAULT_TABLE table=test-table [id=e8403ee1fdb14b488f38b61a88534112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:16.192982 27343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd6d2314599b4a87b994b9d99ace18d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:16.193037 27658 tablet_bootstrap.cc:492] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea: No bootstrap required, opened a new log
I20260808 04:35:16.193117 27658 ts_tablet_manager.cc:1397] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260808 04:35:16.193660 27658 raft_consensus.cc:348] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.193809 27658 raft_consensus.cc:374] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:16.193842 27658 raft_consensus.cc:729] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e28fde95f3406499409d4fd6ac31ea, State: Initialized, Role: FOLLOWER
I20260808 04:35:16.193984 27658 consensus_queue.cc:260] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.195201 27658 ts_tablet_manager.cc:1428] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 04:35:16.195864 27660 tablet_bootstrap.cc:492] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Bootstrap starting.
I20260808 04:35:16.196833 27660 tablet_bootstrap.cc:654] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:16.197227 27690 ts_tablet_manager.cc:1918] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:35:16.197359 27690 log.cc:1196] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/wal/wals/ed635a10182847cda2c6ad79cdeeef18
I20260808 04:35:16.197742 27690 ts_tablet_manager.cc:1939] T ed635a10182847cda2c6ad79cdeeef18 P 128cdb84c72d4f34a433ff04613d886d: Deleting consensus metadata
I20260808 04:35:16.199266 26964 catalog_manager.cc:4841] TS 128cdb84c72d4f34a433ff04613d886d (127.15.228.65:38997): tablet ed635a10182847cda2c6ad79cdeeef18 (table test-table [id=75db03d57a73406895e2b3ba6cc065f3]) successfully deleted
I20260808 04:35:16.200325 27662 tablet_bootstrap.cc:492] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d: No bootstrap required, opened a new log
I20260808 04:35:16.200313 27660 tablet_bootstrap.cc:492] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: No bootstrap required, opened a new log
I20260808 04:35:16.200461 27662 ts_tablet_manager.cc:1397] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260808 04:35:16.200477 27660 ts_tablet_manager.cc:1397] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 04:35:16.200922 27662 raft_consensus.cc:348] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.201089 27662 raft_consensus.cc:374] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:16.201161 27662 raft_consensus.cc:729] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128cdb84c72d4f34a433ff04613d886d, State: Initialized, Role: FOLLOWER
I20260808 04:35:16.201134 27660 raft_consensus.cc:348] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.201282 27660 raft_consensus.cc:374] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:16.201324 27660 raft_consensus.cc:729] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf16ea67b6a4b66956c5dc43c0ca4fc, State: Initialized, Role: FOLLOWER
I20260808 04:35:16.201475 27660 consensus_queue.cc:260] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.201339 27662 consensus_queue.cc:260] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.201803 27660 ts_tablet_manager.cc:1428] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260808 04:35:16.202095 27662 ts_tablet_manager.cc:1428] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260808 04:35:16.210414 27624 tablet.cc:2369] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:35:16.410343 27668 raft_consensus.cc:482] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:35:16.410645 27668 raft_consensus.cc:504] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.412309 27668 leader_election.cc:288] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 128cdb84c72d4f34a433ff04613d886d (127.15.228.65:38997), 0cf16ea67b6a4b66956c5dc43c0ca4fc (127.15.228.66:45041)
I20260808 04:35:16.417009 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.412620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:45041 (local address 127.15.228.67:42633)
0808 04:35:16.412982 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:16.412996 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:16.413108 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:16.413374 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:16.413376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:16.413401 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:16.413541 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.413548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.414769 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.414772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.415782 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.415795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.416556 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.416560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.416667 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.416700 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 04:35:16.416778 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 04:35:16.416849 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":75,"threads_started":1}
I20260808 04:35:16.417299 27653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.412742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:42633 (local address 127.15.228.66:45041)
0808 04:35:16.412955 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:16.412960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:16.413092 (+ 132us) server_negotiation.cc:407] Connection header received
0808 04:35:16.413211 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:16.413214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:16.413265 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:16.413344 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:16.413776 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.414567 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.415960 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.416397 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.416504 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.416552 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:16.416878 (+ 326us) server_negotiation.cc:299] Negotiation successful
0808 04:35:16.417035 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260808 04:35:16.417474 27486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.412528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:38997 (local address 127.15.228.67:38179)
0808 04:35:16.412762 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:16.412778 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:16.412885 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:16.413204 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:16.413207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:16.413226 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:16.413412 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.413422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.414314 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.414319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.415455 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.415470 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.416895 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.416900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.417051 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.417085 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 04:35:16.417206 (+ 121us) client_negotiation.cc:240] Negotiation successful
0808 04:35:16.417302 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260808 04:35:16.417482 27652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.412726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:38179 (local address 127.15.228.65:38997)
0808 04:35:16.412863 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:16.412868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:16.412888 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:35:16.413013 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:16.413018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:16.413089 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:16.413172 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:16.413542 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.414113 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.415611 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.416678 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.416814 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.416872 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:16.417238 (+ 366us) server_negotiation.cc:299] Negotiation successful
0808 04:35:16.417319 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:35:16.418054 27370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6d2314599b4a87b994b9d99ace18d5" candidate_uuid: "63e28fde95f3406499409d4fd6ac31ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" is_pre_election: true
I20260808 04:35:16.418144 27191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6d2314599b4a87b994b9d99ace18d5" candidate_uuid: "63e28fde95f3406499409d4fd6ac31ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128cdb84c72d4f34a433ff04613d886d" is_pre_election: true
I20260808 04:35:16.418283 27191 raft_consensus.cc:2454] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63e28fde95f3406499409d4fd6ac31ea in term 0.
I20260808 04:35:16.418357 27370 raft_consensus.cc:2454] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63e28fde95f3406499409d4fd6ac31ea in term 0.
I20260808 04:35:16.418715 27464 leader_election.cc:302] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 128cdb84c72d4f34a433ff04613d886d, 63e28fde95f3406499409d4fd6ac31ea; no voters:
I20260808 04:35:16.419042 27668 raft_consensus.cc:2781] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:35:16.419100 27668 raft_consensus.cc:482] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:35:16.419127 27668 raft_consensus.cc:3037] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:16.420534 27668 raft_consensus.cc:504] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.420917 27668 leader_election.cc:288] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [CANDIDATE]: Term 1 election: Requested vote from peers 128cdb84c72d4f34a433ff04613d886d (127.15.228.65:38997), 0cf16ea67b6a4b66956c5dc43c0ca4fc (127.15.228.66:45041)
I20260808 04:35:16.421360 27191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6d2314599b4a87b994b9d99ace18d5" candidate_uuid: "63e28fde95f3406499409d4fd6ac31ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128cdb84c72d4f34a433ff04613d886d"
I20260808 04:35:16.421365 27370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6d2314599b4a87b994b9d99ace18d5" candidate_uuid: "63e28fde95f3406499409d4fd6ac31ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc"
I20260808 04:35:16.421469 27191 raft_consensus.cc:3037] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:16.421500 27370 raft_consensus.cc:3037] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:16.422434 27191 raft_consensus.cc:2454] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63e28fde95f3406499409d4fd6ac31ea in term 1.
I20260808 04:35:16.422477 27370 raft_consensus.cc:2454] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63e28fde95f3406499409d4fd6ac31ea in term 1.
I20260808 04:35:16.422734 27464 leader_election.cc:302] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 128cdb84c72d4f34a433ff04613d886d, 63e28fde95f3406499409d4fd6ac31ea; no voters:
I20260808 04:35:16.422981 27668 raft_consensus.cc:2781] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:35:16.423211 27668 raft_consensus.cc:686] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [term 1 LEADER]: Becoming Leader. State: Replica: 63e28fde95f3406499409d4fd6ac31ea, State: Running, Role: LEADER
I20260808 04:35:16.423480 27668 consensus_queue.cc:237] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.425909 26988 catalog_manager.cc:5495] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 63e28fde95f3406499409d4fd6ac31ea (127.15.228.67). New cstate: current_term: 1 leader_uuid: "63e28fde95f3406499409d4fd6ac31ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } health_report { overall_health: HEALTHY } } }
W20260808 04:35:16.430586 27249 tablet.cc:2369] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:35:16.441763 27433 tablet.cc:2369] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 04:35:16 dist-test-slave-drl0 krb5kdc[26924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163715, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.126@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM
I20260808 04:35:16.471536 27706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.460959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:44229 (local address 127.0.0.1:46182)
0808 04:35:16.461365 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:16.461378 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:16.461547 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:16.461811 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:16.461814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:16.462255 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:16.462367 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.462373 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.463196 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.463201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.463834 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.463843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.464282 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.464286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.464489 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.465145 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:16.465157 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:16.467073 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:16.469398 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:16.469407 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:16.469413 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:16.469665 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:16.470216 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:16.470227 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:16.470229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:16.470331 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:16.471022 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:16.471029 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:16.471142 (+ 113us) client_negotiation.cc:769] Sending connection context
0808 04:35:16.471249 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 04:35:16.471388 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":40,"thread_start_us":133,"threads_started":1}
I20260808 04:35:16.471957 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.461082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.15.228.126:44229)
0808 04:35:16.461293 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:16.461299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:16.461478 (+ 179us) server_negotiation.cc:407] Connection header received
0808 04:35:16.461649 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:16.461652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:16.461699 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:16.461788 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:16.462494 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.463087 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.463951 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.464165 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.464275 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.467238 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:16.467241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:16.467243 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:16.467265 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:16.469257 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:16.469854 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:16.469860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:16.469862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:16.470044 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:16.470462 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:16.470465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:16.470466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:16.470678 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:16.471018 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:16.471437 (+ 419us) server_negotiation.cc:299] Negotiation successful
0808 04:35:16.471751 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":99}
I20260808 04:35:16.477252 26988 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:46182:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 04:35:16.478531 26988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:35:16.485565 27164 tablet_service.cc:1467] Processing CreateTablet for tablet aef03099c280483783e04293fe3ba0c2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8377040849034f768fb7fa879898cd6c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:35:16.485565 27343 tablet_service.cc:1467] Processing CreateTablet for tablet aef03099c280483783e04293fe3ba0c2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8377040849034f768fb7fa879898cd6c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:35:16.486013 27164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef03099c280483783e04293fe3ba0c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:16.486012 27343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef03099c280483783e04293fe3ba0c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:16.485832 27534 tablet_service.cc:1467] Processing CreateTablet for tablet aef03099c280483783e04293fe3ba0c2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8377040849034f768fb7fa879898cd6c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 04:35:16.486385 27534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef03099c280483783e04293fe3ba0c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:16.489125 27662 tablet_bootstrap.cc:492] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d: Bootstrap starting.
I20260808 04:35:16.489271 27660 tablet_bootstrap.cc:492] T aef03099c280483783e04293fe3ba0c2 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Bootstrap starting.
I20260808 04:35:16.489969 27662 tablet_bootstrap.cc:654] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:16.490175 27660 tablet_bootstrap.cc:654] T aef03099c280483783e04293fe3ba0c2 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:16.490175 27658 tablet_bootstrap.cc:492] T aef03099c280483783e04293fe3ba0c2 P 63e28fde95f3406499409d4fd6ac31ea: Bootstrap starting.
I20260808 04:35:16.491261 27658 tablet_bootstrap.cc:654] T aef03099c280483783e04293fe3ba0c2 P 63e28fde95f3406499409d4fd6ac31ea: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:16.491418 27662 tablet_bootstrap.cc:492] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d: No bootstrap required, opened a new log
I20260808 04:35:16.491637 27662 ts_tablet_manager.cc:1397] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:35:16.492169 27662 raft_consensus.cc:348] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.492295 27662 raft_consensus.cc:374] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:16.492357 27662 raft_consensus.cc:729] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128cdb84c72d4f34a433ff04613d886d, State: Initialized, Role: FOLLOWER
I20260808 04:35:16.492579 27662 consensus_queue.cc:260] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.492764 27658 tablet_bootstrap.cc:492] T aef03099c280483783e04293fe3ba0c2 P 63e28fde95f3406499409d4fd6ac31ea: No bootstrap required, opened a new log
I20260808 04:35:16.492859 27658 ts_tablet_manager.cc:1397] T aef03099c280483783e04293fe3ba0c2 P 63e28fde95f3406499409d4fd6ac31ea: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:35:16.492838 27667 tablet_replica.cc:439] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } } }
I20260808 04:35:16.493040 27667 tablet_replica.cc:442] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:35:16.493150 27660 tablet_bootstrap.cc:492] T aef03099c280483783e04293fe3ba0c2 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: No bootstrap required, opened a new log
I20260808 04:35:16.493224 27660 ts_tablet_manager.cc:1397] T aef03099c280483783e04293fe3ba0c2 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 04:35:16.493341 27658 raft_consensus.cc:348] T aef03099c280483783e04293fe3ba0c2 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.493453 27658 raft_consensus.cc:374] T aef03099c280483783e04293fe3ba0c2 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:16.493485 27658 raft_consensus.cc:729] T aef03099c280483783e04293fe3ba0c2 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e28fde95f3406499409d4fd6ac31ea, State: Initialized, Role: FOLLOWER
I20260808 04:35:16.493619 27658 consensus_queue.cc:260] T aef03099c280483783e04293fe3ba0c2 P 63e28fde95f3406499409d4fd6ac31ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.493651 27660 raft_consensus.cc:348] T aef03099c280483783e04293fe3ba0c2 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.493779 27660 raft_consensus.cc:374] T aef03099c280483783e04293fe3ba0c2 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:16.493793 27662 ts_tablet_manager.cc:1428] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 04:35:16.493818 27660 raft_consensus.cc:729] T aef03099c280483783e04293fe3ba0c2 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf16ea67b6a4b66956c5dc43c0ca4fc, State: Initialized, Role: FOLLOWER
I20260808 04:35:16.493902 27658 ts_tablet_manager.cc:1428] T aef03099c280483783e04293fe3ba0c2 P 63e28fde95f3406499409d4fd6ac31ea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:35:16.493957 27660 consensus_queue.cc:260] T aef03099c280483783e04293fe3ba0c2 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.494038 27668 tablet_replica.cc:439] T aef03099c280483783e04293fe3ba0c2 P 63e28fde95f3406499409d4fd6ac31ea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } } }
I20260808 04:35:16.494154 27668 tablet_replica.cc:442] T aef03099c280483783e04293fe3ba0c2 P 63e28fde95f3406499409d4fd6ac31ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:35:16.494251 27660 ts_tablet_manager.cc:1428] T aef03099c280483783e04293fe3ba0c2 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:35:16.494451 27683 tablet_replica.cc:439] T aef03099c280483783e04293fe3ba0c2 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } } }
I20260808 04:35:16.494594 27683 tablet_replica.cc:442] T aef03099c280483783e04293fe3ba0c2 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:35:16.669374 27667 raft_consensus.cc:482] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:35:16.669540 27667 raft_consensus.cc:504] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.671290 27667 leader_election.cc:288] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf16ea67b6a4b66956c5dc43c0ca4fc (127.15.228.66:45041), 63e28fde95f3406499409d4fd6ac31ea (127.15.228.67:37655)
I20260808 04:35:16.675321 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.671723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:53489 (local address 127.15.228.67:37655)
0808 04:35:16.671962 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:16.671967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:16.672072 (+ 105us) server_negotiation.cc:407] Connection header received
0808 04:35:16.672237 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:16.672240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:16.672291 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:16.672390 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:16.672672 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.673193 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.674268 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.674683 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.674813 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.674865 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:16.675110 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 04:35:16.675181 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":82,"threads_started":1}
I20260808 04:35:16.675518 27653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.671632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:44911 (local address 127.15.228.66:45041)
0808 04:35:16.671785 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:16.671790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:16.671927 (+ 137us) server_negotiation.cc:407] Connection header received
0808 04:35:16.672089 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:16.672093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:16.672140 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:16.672221 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:16.672520 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.673038 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.674220 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.674623 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.674769 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.674833 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:16.675053 (+ 220us) server_negotiation.cc:299] Negotiation successful
0808 04:35:16.675127 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:35:16.675321 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.671626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:37655 (local address 127.15.228.65:53489)
0808 04:35:16.671964 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:16.671979 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:16.672130 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:16.672411 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:16.672414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:16.672431 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:16.672562 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.672568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.673327 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.673330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.674140 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.674149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.674814 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.674818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.674918 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.674995 (+ 77us) client_negotiation.cc:769] Sending connection context
0808 04:35:16.675091 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 04:35:16.675162 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":78,"threads_started":1}
I20260808 04:35:16.675549 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.671500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:45041 (local address 127.15.228.65:44911)
0808 04:35:16.671804 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:16.671835 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:16.671969 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:16.672247 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:16.672250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:16.672267 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:16.672410 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.672416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.673161 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.673165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.674090 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.674100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.674766 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.674770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.674895 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.674916 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 04:35:16.675037 (+ 121us) client_negotiation.cc:240] Negotiation successful
0808 04:35:16.675110 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":55,"thread_start_us":72,"threads_started":1}
I20260808 04:35:16.676307 27561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef03099c280483783e04293fe3ba0c2" candidate_uuid: "128cdb84c72d4f34a433ff04613d886d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63e28fde95f3406499409d4fd6ac31ea" is_pre_election: true
I20260808 04:35:16.676492 27561 raft_consensus.cc:2454] T aef03099c280483783e04293fe3ba0c2 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 128cdb84c72d4f34a433ff04613d886d in term 0.
I20260808 04:35:16.676959 27103 leader_election.cc:302] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 128cdb84c72d4f34a433ff04613d886d, 63e28fde95f3406499409d4fd6ac31ea; no voters:
I20260808 04:35:16.677007 27370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef03099c280483783e04293fe3ba0c2" candidate_uuid: "128cdb84c72d4f34a433ff04613d886d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" is_pre_election: true
I20260808 04:35:16.677287 27370 raft_consensus.cc:2454] T aef03099c280483783e04293fe3ba0c2 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 128cdb84c72d4f34a433ff04613d886d in term 0.
I20260808 04:35:16.677299 27667 raft_consensus.cc:2781] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:35:16.677383 27667 raft_consensus.cc:482] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:35:16.677424 27667 raft_consensus.cc:3037] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:16.678916 27667 raft_consensus.cc:504] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.679554 27667 leader_election.cc:288] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d [CANDIDATE]: Term 1 election: Requested vote from peers 0cf16ea67b6a4b66956c5dc43c0ca4fc (127.15.228.66:45041), 63e28fde95f3406499409d4fd6ac31ea (127.15.228.67:37655)
I20260808 04:35:16.680019 27561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef03099c280483783e04293fe3ba0c2" candidate_uuid: "128cdb84c72d4f34a433ff04613d886d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63e28fde95f3406499409d4fd6ac31ea"
I20260808 04:35:16.680019 27370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef03099c280483783e04293fe3ba0c2" candidate_uuid: "128cdb84c72d4f34a433ff04613d886d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc"
I20260808 04:35:16.680167 27370 raft_consensus.cc:3037] T aef03099c280483783e04293fe3ba0c2 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:16.680166 27561 raft_consensus.cc:3037] T aef03099c280483783e04293fe3ba0c2 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:16.681515 27370 raft_consensus.cc:2454] T aef03099c280483783e04293fe3ba0c2 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 128cdb84c72d4f34a433ff04613d886d in term 1.
I20260808 04:35:16.681515 27561 raft_consensus.cc:2454] T aef03099c280483783e04293fe3ba0c2 P 63e28fde95f3406499409d4fd6ac31ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 128cdb84c72d4f34a433ff04613d886d in term 1.
I20260808 04:35:16.681874 27102 leader_election.cc:302] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0cf16ea67b6a4b66956c5dc43c0ca4fc, 128cdb84c72d4f34a433ff04613d886d, 63e28fde95f3406499409d4fd6ac31ea; no voters:
I20260808 04:35:16.682135 27667 raft_consensus.cc:2781] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:35:16.682379 27667 raft_consensus.cc:686] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d [term 1 LEADER]: Becoming Leader. State: Replica: 128cdb84c72d4f34a433ff04613d886d, State: Running, Role: LEADER
I20260808 04:35:16.682663 27667 consensus_queue.cc:237] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.684523 27715 tablet_replica.cc:439] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d: TxnStatusTablet state changed. Reason: New leader 128cdb84c72d4f34a433ff04613d886d. Latest consensus state: current_term: 1 leader_uuid: "128cdb84c72d4f34a433ff04613d886d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } } }
I20260808 04:35:16.684653 27715 tablet_replica.cc:442] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:35:16.685134 27717 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 04:35:16.685292 27717 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 04:35:16.686512 26983 catalog_manager.cc:5495] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d reported cstate change: term changed from 0 to 1, leader changed from <none> to 128cdb84c72d4f34a433ff04613d886d (127.15.228.65). New cstate: current_term: 1 leader_uuid: "128cdb84c72d4f34a433ff04613d886d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } health_report { overall_health: UNKNOWN } } }
I20260808 04:35:16.736753 27706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.731339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:38997 (local address 127.0.0.1:47194)
0808 04:35:16.731531 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:16.731552 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:16.731686 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:16.732150 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:16.732153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:16.732166 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:35:16.732319 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.732326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.733506 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.733510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.734302 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.734312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.735184 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.735187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.735259 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.735310 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:35:16.736436 (+ 1126us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:35:16.736453 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 04:35:16.736528 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 04:35:16.736607 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 04:35:16.736845 27652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.731526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.15.228.65:38997)
0808 04:35:16.731689 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:16.731694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:16.731712 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:35:16.731835 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:16.731838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:16.731908 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:16.732017 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:35:16.732484 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.733344 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.734516 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.734961 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.735198 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.735464 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:35:16.736266 (+ 802us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:35:16.736451 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:16.736551 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 04:35:16.736649 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:35:16.740913 27561 raft_consensus.cc:1261] T aef03099c280483783e04293fe3ba0c2 P 63e28fde95f3406499409d4fd6ac31ea [term 1 FOLLOWER]: Refusing update from remote peer 128cdb84c72d4f34a433ff04613d886d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:35:16.741027 27370 raft_consensus.cc:1261] T aef03099c280483783e04293fe3ba0c2 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 1 FOLLOWER]: Refusing update from remote peer 128cdb84c72d4f34a433ff04613d886d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:35:16.741609 27667 consensus_queue.cc:1035] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:35:16.741958 27715 consensus_queue.cc:1035] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:35:16.743544 27668 tablet_replica.cc:439] T aef03099c280483783e04293fe3ba0c2 P 63e28fde95f3406499409d4fd6ac31ea: TxnStatusTablet state changed. Reason: New leader 128cdb84c72d4f34a433ff04613d886d. Latest consensus state: current_term: 1 leader_uuid: "128cdb84c72d4f34a433ff04613d886d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } } }
I20260808 04:35:16.743686 27668 tablet_replica.cc:442] T aef03099c280483783e04293fe3ba0c2 P 63e28fde95f3406499409d4fd6ac31ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:35:16.743963 27683 tablet_replica.cc:439] T aef03099c280483783e04293fe3ba0c2 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: TxnStatusTablet state changed. Reason: New leader 128cdb84c72d4f34a433ff04613d886d. Latest consensus state: current_term: 1 leader_uuid: "128cdb84c72d4f34a433ff04613d886d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } } }
I20260808 04:35:16.744091 27683 tablet_replica.cc:442] T aef03099c280483783e04293fe3ba0c2 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:35:16.746018 27721 mvcc.cc:205] Tried to move back new op lower bound from 7316126583762219008 to 7316126583534833664. Current Snapshot: MvccSnapshot[applied={T|T < 7316126583762219008}]
I20260808 04:35:16.746140 27667 tablet_replica.cc:439] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "128cdb84c72d4f34a433ff04613d886d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } } }
I20260808 04:35:16.746263 27667 tablet_replica.cc:442] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:35:16.748011 27668 tablet_replica.cc:439] T aef03099c280483783e04293fe3ba0c2 P 63e28fde95f3406499409d4fd6ac31ea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "128cdb84c72d4f34a433ff04613d886d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } } }
I20260808 04:35:16.748114 27668 tablet_replica.cc:442] T aef03099c280483783e04293fe3ba0c2 P 63e28fde95f3406499409d4fd6ac31ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:35:16.750337 27715 tablet_replica.cc:439] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "128cdb84c72d4f34a433ff04613d886d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } } }
I20260808 04:35:16.750447 27715 tablet_replica.cc:442] T aef03099c280483783e04293fe3ba0c2 P 128cdb84c72d4f34a433ff04613d886d: This TxnStatusTablet replica's current role is: LEADER
I20260808 04:35:16.752755 27683 tablet_replica.cc:439] T aef03099c280483783e04293fe3ba0c2 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "128cdb84c72d4f34a433ff04613d886d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } } }
I20260808 04:35:16.752878 27683 tablet_replica.cc:442] T aef03099c280483783e04293fe3ba0c2 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 04:35:16.759828 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.754128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.15.228.67:37655)
0808 04:35:16.754234 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:16.754240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:16.754672 (+ 432us) server_negotiation.cc:407] Connection header received
0808 04:35:16.754910 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:16.754913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:16.755015 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:16.755095 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:35:16.755667 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.756352 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.757920 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.758137 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.758229 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.758789 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:35:16.759344 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:35:16.759453 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:16.759595 (+ 142us) server_negotiation.cc:299] Negotiation successful
0808 04:35:16.759648 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 04:35:16.772122 27736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.767507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:38997 (local address 127.15.228.67:38397)
0808 04:35:16.767917 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:16.767934 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:16.768049 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:16.768584 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:16.768589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:16.768600 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:35:16.768739 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.768749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.769989 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.769991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.770674 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.770684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.771313 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.771316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.771387 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.771455 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:35:16.771841 (+ 386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:35:16.771848 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 04:35:16.771912 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 04:35:16.771979 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":94,"threads_started":1}
I20260808 04:35:16.772146 27652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.767570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:38397 (local address 127.15.228.65:38997)
0808 04:35:16.767716 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:16.767720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:16.768033 (+ 313us) server_negotiation.cc:407] Connection header received
0808 04:35:16.768193 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:16.768196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:16.768335 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:16.768432 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:35:16.769066 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.769863 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.770794 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.771151 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.771263 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.771604 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:35:16.771688 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:35:16.771765 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:16.771954 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 04:35:16.772015 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:35:16.780196 27370 raft_consensus.cc:1261] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 1 FOLLOWER]: Refusing update from remote peer 63e28fde95f3406499409d4fd6ac31ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:35:16.780290 27191 raft_consensus.cc:1261] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d [term 1 FOLLOWER]: Refusing update from remote peer 63e28fde95f3406499409d4fd6ac31ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:35:16.780767 27696 consensus_queue.cc:1035] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:35:16.780965 27668 consensus_queue.cc:1035] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:35:16.807512 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.802548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:50353 (local address 127.15.228.67:37655)
0808 04:35:16.802691 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:16.802697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:16.803398 (+ 701us) server_negotiation.cc:407] Connection header received
0808 04:35:16.803677 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:16.803681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:16.803748 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:16.803798 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:35:16.804341 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.804869 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.806115 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.806482 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.806558 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.807013 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:35:16.807101 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:35:16.807146 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:16.807301 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 04:35:16.807361 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 04:35:16.808207 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.802514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:37655 (local address 127.15.228.65:50353)
0808 04:35:16.803221 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:16.803246 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:16.803525 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:16.803951 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:16.803953 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:16.804055 (+ 102us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:35:16.804203 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.804213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.804995 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.804998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.805969 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:16.805981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.806677 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.806680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:16.806760 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.806810 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:35:16.807208 (+ 398us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:35:16.807214 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 04:35:16.807951 (+ 737us) client_negotiation.cc:240] Negotiation successful
0808 04:35:16.808044 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":143,"threads_started":1}
I20260808 04:35:16.837330 26983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:35:16.837540 26983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:35:16.840230 26983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6: Sending DeleteTablet for 3 replicas of tablet cd6d2314599b4a87b994b9d99ace18d5
I20260808 04:35:16.840905 27164 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6d2314599b4a87b994b9d99ace18d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:16 UTC) from {username='oryx', principal='oryx/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:47182
I20260808 04:35:16.841035 27534 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6d2314599b4a87b994b9d99ace18d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:16 UTC) from {username='oryx', principal='oryx/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:49568
I20260808 04:35:16.841485 27748 tablet_replica.cc:330] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d: stopping tablet replica
I20260808 04:35:16.841653 27748 raft_consensus.cc:2229] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:35:16.841801 27748 raft_consensus.cc:2258] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:16.841995 27343 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6d2314599b4a87b994b9d99ace18d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:16 UTC) from {username='oryx', principal='oryx/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:45368
I20260808 04:35:16.842546 27750 tablet_replica.cc:330] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: stopping tablet replica
I20260808 04:35:16.842695 27750 raft_consensus.cc:2229] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:35:16.842834 27750 raft_consensus.cc:2258] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:16.843132 27749 tablet_replica.cc:330] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea: stopping tablet replica
I20260808 04:35:16.843282 27749 raft_consensus.cc:2229] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:35:16.843636 27749 raft_consensus.cc:2258] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:16.844292 27750 ts_tablet_manager.cc:1905] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:16.847584 27748 ts_tablet_manager.cc:1905] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:16.847735 27749 ts_tablet_manager.cc:1905] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:16.850871 27748 ts_tablet_manager.cc:1918] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:35:16.850999 27748 log.cc:1196] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/wal/wals/cd6d2314599b4a87b994b9d99ace18d5
I20260808 04:35:16.851325 27748 ts_tablet_manager.cc:1939] T cd6d2314599b4a87b994b9d99ace18d5 P 128cdb84c72d4f34a433ff04613d886d: Deleting consensus metadata
I20260808 04:35:16.851701 27750 ts_tablet_manager.cc:1918] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:35:16.851796 27750 log.cc:1196] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/cd6d2314599b4a87b994b9d99ace18d5
I20260808 04:35:16.852064 27750 ts_tablet_manager.cc:1939] T cd6d2314599b4a87b994b9d99ace18d5 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Deleting consensus metadata
I20260808 04:35:16.852352 26964 catalog_manager.cc:4841] TS 128cdb84c72d4f34a433ff04613d886d (127.15.228.65:38997): tablet cd6d2314599b4a87b994b9d99ace18d5 (table test-table [id=e8403ee1fdb14b488f38b61a88534112]) successfully deleted
I20260808 04:35:16.852972 26966 catalog_manager.cc:4841] TS 0cf16ea67b6a4b66956c5dc43c0ca4fc (127.15.228.66:45041): tablet cd6d2314599b4a87b994b9d99ace18d5 (table test-table [id=e8403ee1fdb14b488f38b61a88534112]) successfully deleted
I20260808 04:35:16.854532 27749 ts_tablet_manager.cc:1918] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:35:16.854763 27749 log.cc:1196] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/cd6d2314599b4a87b994b9d99ace18d5
I20260808 04:35:16.855188 27749 ts_tablet_manager.cc:1939] T cd6d2314599b4a87b994b9d99ace18d5 P 63e28fde95f3406499409d4fd6ac31ea: Deleting consensus metadata
I20260808 04:35:16.856016 26967 catalog_manager.cc:4841] TS 63e28fde95f3406499409d4fd6ac31ea (127.15.228.67:37655): tablet cd6d2314599b4a87b994b9d99ace18d5 (table test-table [id=e8403ee1fdb14b488f38b61a88534112]) successfully deleted
Aug 08 04:35:16 dist-test-slave-drl0 krb5kdc[26924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 04:35:16 dist-test-slave-drl0 krb5kdc[26924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:16 dist-test-slave-drl0 krb5kdc[26924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 04:35:16.893392 27042 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:16.883669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46196 (local address 127.15.228.126:44229)
0808 04:35:16.883812 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:16.883817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:16.883834 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:35:16.883958 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:16.883962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:16.884012 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:16.884113 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:16.885039 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.885931 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.887216 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.887625 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.887763 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.893070 (+ 5307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46196: BlockingRecv error: recv got EOF from 127.0.0.1:46196 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 08 04:35:16 dist-test-slave-drl0 krb5kdc[26924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM
I20260808 04:35:16.905725 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.894864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46204 (local address 127.15.228.126:44229)
0808 04:35:16.895012 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:16.895017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:16.895580 (+ 563us) server_negotiation.cc:407] Connection header received
0808 04:35:16.895956 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:16.895959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:16.896012 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:16.896103 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:16.897093 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.897842 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.899060 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.899345 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.899448 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.902439 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:16.902445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:16.902453 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:16.902478 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:16.903996 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:16.904452 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:16.904461 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:16.904462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:16.904536 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:16.904828 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:16.904831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:16.904833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:16.905025 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:16.905178 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:16.905425 (+ 247us) server_negotiation.cc:299] Negotiation successful
0808 04:35:16.905544 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:35:16.908815 26983 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:35:16.909703 26983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:35:16.916950 27164 tablet_service.cc:1467] Processing CreateTablet for tablet a344e4cea9844e1182493fc85bc4a4c3 (DEFAULT_TABLE table=test-table [id=94fe32a9124142ad9a3d5bf35eac1728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:16.917166 27343 tablet_service.cc:1467] Processing CreateTablet for tablet a344e4cea9844e1182493fc85bc4a4c3 (DEFAULT_TABLE table=test-table [id=94fe32a9124142ad9a3d5bf35eac1728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:16.917243 27534 tablet_service.cc:1467] Processing CreateTablet for tablet a344e4cea9844e1182493fc85bc4a4c3 (DEFAULT_TABLE table=test-table [id=94fe32a9124142ad9a3d5bf35eac1728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:16.917469 27164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a344e4cea9844e1182493fc85bc4a4c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:16.917491 27343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a344e4cea9844e1182493fc85bc4a4c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:16.917582 27534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a344e4cea9844e1182493fc85bc4a4c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:16.920933 27658 tablet_bootstrap.cc:492] T a344e4cea9844e1182493fc85bc4a4c3 P 63e28fde95f3406499409d4fd6ac31ea: Bootstrap starting.
I20260808 04:35:16.921063 27660 tablet_bootstrap.cc:492] T a344e4cea9844e1182493fc85bc4a4c3 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Bootstrap starting.
I20260808 04:35:16.921873 27660 tablet_bootstrap.cc:654] T a344e4cea9844e1182493fc85bc4a4c3 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:16.921975 27658 tablet_bootstrap.cc:654] T a344e4cea9844e1182493fc85bc4a4c3 P 63e28fde95f3406499409d4fd6ac31ea: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:16.923161 27660 tablet_bootstrap.cc:492] T a344e4cea9844e1182493fc85bc4a4c3 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: No bootstrap required, opened a new log
I20260808 04:35:16.923259 27660 ts_tablet_manager.cc:1397] T a344e4cea9844e1182493fc85bc4a4c3 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 04:35:16.923708 27660 raft_consensus.cc:348] T a344e4cea9844e1182493fc85bc4a4c3 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.923831 27660 raft_consensus.cc:374] T a344e4cea9844e1182493fc85bc4a4c3 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:16.923868 27660 raft_consensus.cc:729] T a344e4cea9844e1182493fc85bc4a4c3 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf16ea67b6a4b66956c5dc43c0ca4fc, State: Initialized, Role: FOLLOWER
I20260808 04:35:16.924026 27660 consensus_queue.cc:260] T a344e4cea9844e1182493fc85bc4a4c3 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.924291 27660 ts_tablet_manager.cc:1428] T a344e4cea9844e1182493fc85bc4a4c3 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 04:35:16.924623 27662 tablet_bootstrap.cc:492] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d: Bootstrap starting.
I20260808 04:35:16.925577 27662 tablet_bootstrap.cc:654] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:16.927461 27658 tablet_bootstrap.cc:492] T a344e4cea9844e1182493fc85bc4a4c3 P 63e28fde95f3406499409d4fd6ac31ea: No bootstrap required, opened a new log
I20260808 04:35:16.927568 27658 ts_tablet_manager.cc:1397] T a344e4cea9844e1182493fc85bc4a4c3 P 63e28fde95f3406499409d4fd6ac31ea: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260808 04:35:16.928081 27658 raft_consensus.cc:348] T a344e4cea9844e1182493fc85bc4a4c3 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.928227 27658 raft_consensus.cc:374] T a344e4cea9844e1182493fc85bc4a4c3 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:16.928270 27658 raft_consensus.cc:729] T a344e4cea9844e1182493fc85bc4a4c3 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e28fde95f3406499409d4fd6ac31ea, State: Initialized, Role: FOLLOWER
I20260808 04:35:16.928447 27658 consensus_queue.cc:260] T a344e4cea9844e1182493fc85bc4a4c3 P 63e28fde95f3406499409d4fd6ac31ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.928790 27658 ts_tablet_manager.cc:1428] T a344e4cea9844e1182493fc85bc4a4c3 P 63e28fde95f3406499409d4fd6ac31ea: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 04:35:16.929721 27662 tablet_bootstrap.cc:492] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d: No bootstrap required, opened a new log
I20260808 04:35:16.929831 27662 ts_tablet_manager.cc:1397] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 04:35:16.930320 27662 raft_consensus.cc:348] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.930452 27662 raft_consensus.cc:374] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:16.930485 27662 raft_consensus.cc:729] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128cdb84c72d4f34a433ff04613d886d, State: Initialized, Role: FOLLOWER
I20260808 04:35:16.930624 27662 consensus_queue.cc:260] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.930994 27662 ts_tablet_manager.cc:1428] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260808 04:35:16.933410 27667 raft_consensus.cc:482] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:35:16.933508 27667 raft_consensus.cc:504] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.934430 27370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a344e4cea9844e1182493fc85bc4a4c3" candidate_uuid: "128cdb84c72d4f34a433ff04613d886d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" is_pre_election: true
I20260808 04:35:16.934597 27370 raft_consensus.cc:2454] T a344e4cea9844e1182493fc85bc4a4c3 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 128cdb84c72d4f34a433ff04613d886d in term 0.
I20260808 04:35:16.933966 27667 leader_election.cc:288] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf16ea67b6a4b66956c5dc43c0ca4fc (127.15.228.66:45041), 63e28fde95f3406499409d4fd6ac31ea (127.15.228.67:37655)
I20260808 04:35:16.934996 27102 leader_election.cc:302] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf16ea67b6a4b66956c5dc43c0ca4fc, 128cdb84c72d4f34a433ff04613d886d; no voters:
I20260808 04:35:16.935175 27667 raft_consensus.cc:2781] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:35:16.935236 27667 raft_consensus.cc:482] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:35:16.935274 27667 raft_consensus.cc:3037] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:16.936416 27561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a344e4cea9844e1182493fc85bc4a4c3" candidate_uuid: "128cdb84c72d4f34a433ff04613d886d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63e28fde95f3406499409d4fd6ac31ea" is_pre_election: true
I20260808 04:35:16.937366 27667 raft_consensus.cc:504] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.938421 27370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a344e4cea9844e1182493fc85bc4a4c3" candidate_uuid: "128cdb84c72d4f34a433ff04613d886d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc"
I20260808 04:35:16.938602 27370 raft_consensus.cc:3037] T a344e4cea9844e1182493fc85bc4a4c3 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:16.936686 27561 raft_consensus.cc:2454] T a344e4cea9844e1182493fc85bc4a4c3 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 128cdb84c72d4f34a433ff04613d886d in term 0.
I20260808 04:35:16.940374 27370 raft_consensus.cc:2454] T a344e4cea9844e1182493fc85bc4a4c3 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 128cdb84c72d4f34a433ff04613d886d in term 1.
I20260808 04:35:16.940989 27102 leader_election.cc:302] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf16ea67b6a4b66956c5dc43c0ca4fc, 128cdb84c72d4f34a433ff04613d886d; no voters:
I20260808 04:35:16.941087 27667 leader_election.cc:288] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [CANDIDATE]: Term 1 election: Requested vote from peers 0cf16ea67b6a4b66956c5dc43c0ca4fc (127.15.228.66:45041), 63e28fde95f3406499409d4fd6ac31ea (127.15.228.67:37655)
I20260808 04:35:16.941154 27561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a344e4cea9844e1182493fc85bc4a4c3" candidate_uuid: "128cdb84c72d4f34a433ff04613d886d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63e28fde95f3406499409d4fd6ac31ea"
I20260808 04:35:16.941260 27667 raft_consensus.cc:2781] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:35:16.941309 27561 raft_consensus.cc:3037] T a344e4cea9844e1182493fc85bc4a4c3 P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:16.941355 27667 raft_consensus.cc:686] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [term 1 LEADER]: Becoming Leader. State: Replica: 128cdb84c72d4f34a433ff04613d886d, State: Running, Role: LEADER
I20260808 04:35:16.941547 27667 consensus_queue.cc:237] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:16.943210 27561 raft_consensus.cc:2454] T a344e4cea9844e1182493fc85bc4a4c3 P 63e28fde95f3406499409d4fd6ac31ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 128cdb84c72d4f34a433ff04613d886d in term 1.
I20260808 04:35:16.944741 26983 catalog_manager.cc:5495] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d reported cstate change: term changed from 0 to 1, leader changed from <none> to 128cdb84c72d4f34a433ff04613d886d (127.15.228.65). New cstate: current_term: 1 leader_uuid: "128cdb84c72d4f34a433ff04613d886d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } health_report { overall_health: UNKNOWN } } }
I20260808 04:35:16.959012 27652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:16.954377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47210 (local address 127.15.228.65:38997)
0808 04:35:16.954516 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:16.954520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:16.954534 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:35:16.954629 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:16.954632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:16.954675 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:16.954732 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:35:16.955203 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.955787 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.957538 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:16.957880 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:16.958019 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:16.958418 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:35:16.958509 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:35:16.958588 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:16.958764 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 04:35:16.958821 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:35:16.961651 27561 raft_consensus.cc:1261] T a344e4cea9844e1182493fc85bc4a4c3 P 63e28fde95f3406499409d4fd6ac31ea [term 1 FOLLOWER]: Refusing update from remote peer 128cdb84c72d4f34a433ff04613d886d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:35:16.961731 27370 raft_consensus.cc:1261] T a344e4cea9844e1182493fc85bc4a4c3 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 1 FOLLOWER]: Refusing update from remote peer 128cdb84c72d4f34a433ff04613d886d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:35:16.962448 27738 consensus_queue.cc:1035] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:35:16.962865 27738 consensus_queue.cc:1035] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:35:16.980409 26983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:35:16.980579 26983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:35:16.983325 26983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6: Sending DeleteTablet for 3 replicas of tablet a344e4cea9844e1182493fc85bc4a4c3
I20260808 04:35:16.983855 27164 tablet_service.cc:1514] Processing DeleteTablet for tablet a344e4cea9844e1182493fc85bc4a4c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:16 UTC) from {username='oryx', principal='oryx/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:47182
I20260808 04:35:16.984109 27748 tablet_replica.cc:330] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d: stopping tablet replica
I20260808 04:35:16.984236 27748 raft_consensus.cc:2229] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:35:16.984553 27748 raft_consensus.cc:2258] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:16.984745 27343 tablet_service.cc:1514] Processing DeleteTablet for tablet a344e4cea9844e1182493fc85bc4a4c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:16 UTC) from {username='oryx', principal='oryx/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:45368
I20260808 04:35:16.985136 27750 tablet_replica.cc:330] T a344e4cea9844e1182493fc85bc4a4c3 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: stopping tablet replica
I20260808 04:35:16.985101 26988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 04:35:16.985258 27750 raft_consensus.cc:2229] T a344e4cea9844e1182493fc85bc4a4c3 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:35:16.985379 27750 raft_consensus.cc:2258] T a344e4cea9844e1182493fc85bc4a4c3 P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 04:35:16.985495 26988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:35:16.985538 27534 tablet_service.cc:1514] Processing DeleteTablet for tablet a344e4cea9844e1182493fc85bc4a4c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:16 UTC) from {username='oryx', principal='oryx/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:49568
I20260808 04:35:16.985744 27749 tablet_replica.cc:330] T a344e4cea9844e1182493fc85bc4a4c3 P 63e28fde95f3406499409d4fd6ac31ea: stopping tablet replica
I20260808 04:35:16.985880 27749 raft_consensus.cc:2229] T a344e4cea9844e1182493fc85bc4a4c3 P 63e28fde95f3406499409d4fd6ac31ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:35:16.986025 27749 raft_consensus.cc:2258] T a344e4cea9844e1182493fc85bc4a4c3 P 63e28fde95f3406499409d4fd6ac31ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:16.986671 27748 ts_tablet_manager.cc:1905] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:16.986690 27749 ts_tablet_manager.cc:1905] T a344e4cea9844e1182493fc85bc4a4c3 P 63e28fde95f3406499409d4fd6ac31ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:16.989477 27749 ts_tablet_manager.cc:1918] T a344e4cea9844e1182493fc85bc4a4c3 P 63e28fde95f3406499409d4fd6ac31ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:35:16.989544 27749 log.cc:1196] T a344e4cea9844e1182493fc85bc4a4c3 P 63e28fde95f3406499409d4fd6ac31ea: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/a344e4cea9844e1182493fc85bc4a4c3
I20260808 04:35:16.989576 27748 ts_tablet_manager.cc:1918] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:35:16.989653 27748 log.cc:1196] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/wal/wals/a344e4cea9844e1182493fc85bc4a4c3
I20260808 04:35:16.989796 27749 ts_tablet_manager.cc:1939] T a344e4cea9844e1182493fc85bc4a4c3 P 63e28fde95f3406499409d4fd6ac31ea: Deleting consensus metadata
I20260808 04:35:16.989930 27748 ts_tablet_manager.cc:1939] T a344e4cea9844e1182493fc85bc4a4c3 P 128cdb84c72d4f34a433ff04613d886d: Deleting consensus metadata
I20260808 04:35:16.991518 26967 catalog_manager.cc:4841] TS 63e28fde95f3406499409d4fd6ac31ea (127.15.228.67:37655): tablet a344e4cea9844e1182493fc85bc4a4c3 (table test-table [id=94fe32a9124142ad9a3d5bf35eac1728]) successfully deleted
I20260808 04:35:16.992595 26964 catalog_manager.cc:4841] TS 128cdb84c72d4f34a433ff04613d886d (127.15.228.65:38997): tablet a344e4cea9844e1182493fc85bc4a4c3 (table test-table [id=94fe32a9124142ad9a3d5bf35eac1728]) successfully deleted
I20260808 04:35:16.995085 27164 tablet_service.cc:1467] Processing CreateTablet for tablet ef70415e79c14282ab1ae2f74f1fb3ee (DEFAULT_TABLE table=test-table [id=6822467897e94ad082c791d12dc8ae8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:16.995607 27164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef70415e79c14282ab1ae2f74f1fb3ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:16.999238 27343 tablet_service.cc:1467] Processing CreateTablet for tablet ef70415e79c14282ab1ae2f74f1fb3ee (DEFAULT_TABLE table=test-table [id=6822467897e94ad082c791d12dc8ae8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:16.999562 27343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef70415e79c14282ab1ae2f74f1fb3ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:17.000182 27750 ts_tablet_manager.cc:1905] T a344e4cea9844e1182493fc85bc4a4c3 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:17.002513 27750 ts_tablet_manager.cc:1918] T a344e4cea9844e1182493fc85bc4a4c3 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:35:17.002583 27750 log.cc:1196] T a344e4cea9844e1182493fc85bc4a4c3 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/a344e4cea9844e1182493fc85bc4a4c3
I20260808 04:35:17.002614 27534 tablet_service.cc:1467] Processing CreateTablet for tablet ef70415e79c14282ab1ae2f74f1fb3ee (DEFAULT_TABLE table=test-table [id=6822467897e94ad082c791d12dc8ae8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:17.002874 27750 ts_tablet_manager.cc:1939] T a344e4cea9844e1182493fc85bc4a4c3 P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Deleting consensus metadata
I20260808 04:35:17.002914 27534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef70415e79c14282ab1ae2f74f1fb3ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:17.003759 26966 catalog_manager.cc:4841] TS 0cf16ea67b6a4b66956c5dc43c0ca4fc (127.15.228.66:45041): tablet a344e4cea9844e1182493fc85bc4a4c3 (table test-table [id=94fe32a9124142ad9a3d5bf35eac1728]) successfully deleted
I20260808 04:35:17.004946 27662 tablet_bootstrap.cc:492] T ef70415e79c14282ab1ae2f74f1fb3ee P 128cdb84c72d4f34a433ff04613d886d: Bootstrap starting.
I20260808 04:35:17.005481 27658 tablet_bootstrap.cc:492] T ef70415e79c14282ab1ae2f74f1fb3ee P 63e28fde95f3406499409d4fd6ac31ea: Bootstrap starting.
I20260808 04:35:17.005906 27662 tablet_bootstrap.cc:654] T ef70415e79c14282ab1ae2f74f1fb3ee P 128cdb84c72d4f34a433ff04613d886d: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:17.006666 27660 tablet_bootstrap.cc:492] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Bootstrap starting.
I20260808 04:35:17.007140 27658 tablet_bootstrap.cc:654] T ef70415e79c14282ab1ae2f74f1fb3ee P 63e28fde95f3406499409d4fd6ac31ea: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:17.007551 27660 tablet_bootstrap.cc:654] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:17.009347 27662 tablet_bootstrap.cc:492] T ef70415e79c14282ab1ae2f74f1fb3ee P 128cdb84c72d4f34a433ff04613d886d: No bootstrap required, opened a new log
I20260808 04:35:17.009488 27662 ts_tablet_manager.cc:1397] T ef70415e79c14282ab1ae2f74f1fb3ee P 128cdb84c72d4f34a433ff04613d886d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 04:35:17.009835 27660 tablet_bootstrap.cc:492] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc: No bootstrap required, opened a new log
I20260808 04:35:17.009908 27660 ts_tablet_manager.cc:1397] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 04:35:17.009963 27662 raft_consensus.cc:348] T ef70415e79c14282ab1ae2f74f1fb3ee P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:17.010092 27662 raft_consensus.cc:374] T ef70415e79c14282ab1ae2f74f1fb3ee P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:17.010128 27662 raft_consensus.cc:729] T ef70415e79c14282ab1ae2f74f1fb3ee P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128cdb84c72d4f34a433ff04613d886d, State: Initialized, Role: FOLLOWER
I20260808 04:35:17.010254 27662 consensus_queue.cc:260] T ef70415e79c14282ab1ae2f74f1fb3ee P 128cdb84c72d4f34a433ff04613d886d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:17.010313 27660 raft_consensus.cc:348] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:17.010421 27660 raft_consensus.cc:374] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:17.010452 27660 raft_consensus.cc:729] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf16ea67b6a4b66956c5dc43c0ca4fc, State: Initialized, Role: FOLLOWER
I20260808 04:35:17.010545 27662 ts_tablet_manager.cc:1428] T ef70415e79c14282ab1ae2f74f1fb3ee P 128cdb84c72d4f34a433ff04613d886d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:35:17.010564 27660 consensus_queue.cc:260] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:17.010826 27660 ts_tablet_manager.cc:1428] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:35:17.011631 27658 tablet_bootstrap.cc:492] T ef70415e79c14282ab1ae2f74f1fb3ee P 63e28fde95f3406499409d4fd6ac31ea: No bootstrap required, opened a new log
I20260808 04:35:17.011727 27658 ts_tablet_manager.cc:1397] T ef70415e79c14282ab1ae2f74f1fb3ee P 63e28fde95f3406499409d4fd6ac31ea: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260808 04:35:17.012213 27658 raft_consensus.cc:348] T ef70415e79c14282ab1ae2f74f1fb3ee P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:17.012326 27658 raft_consensus.cc:374] T ef70415e79c14282ab1ae2f74f1fb3ee P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:17.012363 27658 raft_consensus.cc:729] T ef70415e79c14282ab1ae2f74f1fb3ee P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e28fde95f3406499409d4fd6ac31ea, State: Initialized, Role: FOLLOWER
I20260808 04:35:17.012491 27658 consensus_queue.cc:260] T ef70415e79c14282ab1ae2f74f1fb3ee P 63e28fde95f3406499409d4fd6ac31ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:17.012763 27658 ts_tablet_manager.cc:1428] T ef70415e79c14282ab1ae2f74f1fb3ee P 63e28fde95f3406499409d4fd6ac31ea: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 04:35:17.196867 27683 raft_consensus.cc:482] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:35:17.197067 27683 raft_consensus.cc:504] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:17.197625 27683 leader_election.cc:288] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 128cdb84c72d4f34a433ff04613d886d (127.15.228.65:38997), 63e28fde95f3406499409d4fd6ac31ea (127.15.228.67:37655)
I20260808 04:35:17.198231 27191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef70415e79c14282ab1ae2f74f1fb3ee" candidate_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128cdb84c72d4f34a433ff04613d886d" is_pre_election: true
I20260808 04:35:17.198477 27191 raft_consensus.cc:2454] T ef70415e79c14282ab1ae2f74f1fb3ee P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cf16ea67b6a4b66956c5dc43c0ca4fc in term 0.
I20260808 04:35:17.198525 27561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef70415e79c14282ab1ae2f74f1fb3ee" candidate_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63e28fde95f3406499409d4fd6ac31ea" is_pre_election: true
I20260808 04:35:17.198683 27561 raft_consensus.cc:2454] T ef70415e79c14282ab1ae2f74f1fb3ee P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cf16ea67b6a4b66956c5dc43c0ca4fc in term 0.
I20260808 04:35:17.199009 27287 leader_election.cc:302] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf16ea67b6a4b66956c5dc43c0ca4fc, 128cdb84c72d4f34a433ff04613d886d; no voters:
I20260808 04:35:17.199270 27683 raft_consensus.cc:2781] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:35:17.199342 27683 raft_consensus.cc:482] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:35:17.199404 27683 raft_consensus.cc:3037] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:17.200639 27683 raft_consensus.cc:504] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:17.201090 27683 leader_election.cc:288] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [CANDIDATE]: Term 1 election: Requested vote from peers 128cdb84c72d4f34a433ff04613d886d (127.15.228.65:38997), 63e28fde95f3406499409d4fd6ac31ea (127.15.228.67:37655)
I20260808 04:35:17.201538 27191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef70415e79c14282ab1ae2f74f1fb3ee" candidate_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128cdb84c72d4f34a433ff04613d886d"
I20260808 04:35:17.201587 27561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef70415e79c14282ab1ae2f74f1fb3ee" candidate_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63e28fde95f3406499409d4fd6ac31ea"
I20260808 04:35:17.201653 27191 raft_consensus.cc:3037] T ef70415e79c14282ab1ae2f74f1fb3ee P 128cdb84c72d4f34a433ff04613d886d [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:17.201757 27561 raft_consensus.cc:3037] T ef70415e79c14282ab1ae2f74f1fb3ee P 63e28fde95f3406499409d4fd6ac31ea [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:17.202749 27191 raft_consensus.cc:2454] T ef70415e79c14282ab1ae2f74f1fb3ee P 128cdb84c72d4f34a433ff04613d886d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cf16ea67b6a4b66956c5dc43c0ca4fc in term 1.
I20260808 04:35:17.202984 27561 raft_consensus.cc:2454] T ef70415e79c14282ab1ae2f74f1fb3ee P 63e28fde95f3406499409d4fd6ac31ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cf16ea67b6a4b66956c5dc43c0ca4fc in term 1.
I20260808 04:35:17.203154 27287 leader_election.cc:302] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf16ea67b6a4b66956c5dc43c0ca4fc, 128cdb84c72d4f34a433ff04613d886d; no voters:
I20260808 04:35:17.203363 27683 raft_consensus.cc:2781] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:35:17.203544 27683 raft_consensus.cc:686] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 1 LEADER]: Becoming Leader. State: Replica: 0cf16ea67b6a4b66956c5dc43c0ca4fc, State: Running, Role: LEADER
I20260808 04:35:17.203676 27683 consensus_queue.cc:237] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } }
I20260808 04:35:17.205539 26983 catalog_manager.cc:5495] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cf16ea67b6a4b66956c5dc43c0ca4fc (127.15.228.66). New cstate: current_term: 1 leader_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cf16ea67b6a4b66956c5dc43c0ca4fc" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 45041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 } health_report { overall_health: UNKNOWN } } }
I20260808 04:35:17.245963 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:17.242493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45396 (local address 127.15.228.66:45041)
0808 04:35:17.242736 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:17.242740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:17.242752 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:35:17.242790 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:17.242792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:17.242833 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:17.242917 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:35:17.243230 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:17.243832 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:17.244785 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:17.245011 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:17.245094 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:17.245429 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:35:17.245511 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:35:17.245587 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:17.245780 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 04:35:17.245839 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":93,"threads_started":1}
I20260808 04:35:17.258515 27794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:17.253978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:38997 (local address 127.15.228.66:36111)
0808 04:35:17.254286 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:17.254300 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:17.254397 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:17.254859 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:17.254861 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:17.254869 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:35:17.255135 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:17.255155 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:17.256183 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:17.256186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:17.257032 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:17.257043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:17.257610 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:17.257613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:17.257714 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:17.257791 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:35:17.258235 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:35:17.258244 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 04:35:17.258310 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 04:35:17.258385 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":127,"threads_started":1}
I20260808 04:35:17.258522 27652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:17.254161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:36111 (local address 127.15.228.65:38997)
0808 04:35:17.254290 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:17.254294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:17.254393 (+ 99us) server_negotiation.cc:407] Connection header received
0808 04:35:17.254654 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:17.254656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:17.254716 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:17.254797 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:35:17.255392 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:17.256018 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:17.257193 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:17.257433 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:17.257532 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:17.257977 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:35:17.258053 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:35:17.258142 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:17.258364 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 04:35:17.258417 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:35:17.265295 27561 raft_consensus.cc:1261] T ef70415e79c14282ab1ae2f74f1fb3ee P 63e28fde95f3406499409d4fd6ac31ea [term 1 FOLLOWER]: Refusing update from remote peer 0cf16ea67b6a4b66956c5dc43c0ca4fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:35:17.265359 27191 raft_consensus.cc:1261] T ef70415e79c14282ab1ae2f74f1fb3ee P 128cdb84c72d4f34a433ff04613d886d [term 1 FOLLOWER]: Refusing update from remote peer 0cf16ea67b6a4b66956c5dc43c0ca4fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:35:17.265739 27786 consensus_queue.cc:1035] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "63e28fde95f3406499409d4fd6ac31ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 37655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:35:17.265947 27683 consensus_queue.cc:1035] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "128cdb84c72d4f34a433ff04613d886d" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:35:17.287302 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:17.283706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:45041 (local address 127.15.228.65:38435)
0808 04:35:17.283868 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:17.283885 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:17.283967 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:17.284526 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:17.284529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:17.284537 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 04:35:17.284648 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:17.284653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:17.285396 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:17.285401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:17.285987 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:17.285994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:17.286453 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:17.286456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:17.286537 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:17.286591 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 04:35:17.287021 (+ 430us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 04:35:17.287032 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 04:35:17.287107 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 04:35:17.287188 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 04:35:17.287657 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:17.284103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:38435 (local address 127.15.228.66:45041)
0808 04:35:17.284266 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:17.284270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:17.284304 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:35:17.284354 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:17.284359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:17.284412 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:17.284512 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:35:17.284757 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:17.285285 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:17.286110 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:17.286336 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:17.286444 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:17.286768 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:35:17.286857 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:35:17.287008 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:17.287501 (+ 493us) server_negotiation.cc:299] Negotiation successful
0808 04:35:17.287567 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 04:35:17.308564 26988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:35:17.308717 26988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:35:17.310773 26988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c3fa82e61aa4dbbbd8f1d9561f487e6: Sending DeleteTablet for 3 replicas of tablet ef70415e79c14282ab1ae2f74f1fb3ee
I20260808 04:35:17.311435 27164 tablet_service.cc:1514] Processing DeleteTablet for tablet ef70415e79c14282ab1ae2f74f1fb3ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:17 UTC) from {username='oryx', principal='oryx/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:47182
I20260808 04:35:17.311486 27343 tablet_service.cc:1514] Processing DeleteTablet for tablet ef70415e79c14282ab1ae2f74f1fb3ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:17 UTC) from {username='oryx', principal='oryx/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:45368
I20260808 04:35:17.311568 27534 tablet_service.cc:1514] Processing DeleteTablet for tablet ef70415e79c14282ab1ae2f74f1fb3ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:17 UTC) from {username='oryx', principal='oryx/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:49568
I20260808 04:35:17.311651 27750 tablet_replica.cc:330] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc: stopping tablet replica
I20260808 04:35:17.311648 27748 tablet_replica.cc:330] T ef70415e79c14282ab1ae2f74f1fb3ee P 128cdb84c72d4f34a433ff04613d886d: stopping tablet replica
I20260808 04:35:17.311750 27749 tablet_replica.cc:330] T ef70415e79c14282ab1ae2f74f1fb3ee P 63e28fde95f3406499409d4fd6ac31ea: stopping tablet replica
I20260808 04:35:17.311787 27748 raft_consensus.cc:2229] T ef70415e79c14282ab1ae2f74f1fb3ee P 128cdb84c72d4f34a433ff04613d886d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:35:17.311806 27750 raft_consensus.cc:2229] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:35:17.311908 27748 raft_consensus.cc:2258] T ef70415e79c14282ab1ae2f74f1fb3ee P 128cdb84c72d4f34a433ff04613d886d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:17.311885 27749 raft_consensus.cc:2229] T ef70415e79c14282ab1ae2f74f1fb3ee P 63e28fde95f3406499409d4fd6ac31ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:35:17.312023 27749 raft_consensus.cc:2258] T ef70415e79c14282ab1ae2f74f1fb3ee P 63e28fde95f3406499409d4fd6ac31ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:17.312050 27750 raft_consensus.cc:2258] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:17.312681 27748 ts_tablet_manager.cc:1905] T ef70415e79c14282ab1ae2f74f1fb3ee P 128cdb84c72d4f34a433ff04613d886d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:17.312806 27749 ts_tablet_manager.cc:1905] T ef70415e79c14282ab1ae2f74f1fb3ee P 63e28fde95f3406499409d4fd6ac31ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:17.313030 27750 ts_tablet_manager.cc:1905] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:17.315351 27749 ts_tablet_manager.cc:1918] T ef70415e79c14282ab1ae2f74f1fb3ee P 63e28fde95f3406499409d4fd6ac31ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:35:17.315351 27748 ts_tablet_manager.cc:1918] T ef70415e79c14282ab1ae2f74f1fb3ee P 128cdb84c72d4f34a433ff04613d886d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:35:17.315443 27749 log.cc:1196] T ef70415e79c14282ab1ae2f74f1fb3ee P 63e28fde95f3406499409d4fd6ac31ea: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/ef70415e79c14282ab1ae2f74f1fb3ee
I20260808 04:35:17.315477 27748 log.cc:1196] T ef70415e79c14282ab1ae2f74f1fb3ee P 128cdb84c72d4f34a433ff04613d886d: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-0/wal/wals/ef70415e79c14282ab1ae2f74f1fb3ee
I20260808 04:35:17.315745 27749 ts_tablet_manager.cc:1939] T ef70415e79c14282ab1ae2f74f1fb3ee P 63e28fde95f3406499409d4fd6ac31ea: Deleting consensus metadata
I20260808 04:35:17.315780 27748 ts_tablet_manager.cc:1939] T ef70415e79c14282ab1ae2f74f1fb3ee P 128cdb84c72d4f34a433ff04613d886d: Deleting consensus metadata
I20260808 04:35:17.316165 27750 ts_tablet_manager.cc:1918] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 04:35:17.316296 27750 log.cc:1196] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/ef70415e79c14282ab1ae2f74f1fb3ee
I20260808 04:35:17.316514 26964 catalog_manager.cc:4841] TS 128cdb84c72d4f34a433ff04613d886d (127.15.228.65:38997): tablet ef70415e79c14282ab1ae2f74f1fb3ee (table test-table [id=6822467897e94ad082c791d12dc8ae8c]) successfully deleted
I20260808 04:35:17.316689 26967 catalog_manager.cc:4841] TS 63e28fde95f3406499409d4fd6ac31ea (127.15.228.67:37655): tablet ef70415e79c14282ab1ae2f74f1fb3ee (table test-table [id=6822467897e94ad082c791d12dc8ae8c]) successfully deleted
I20260808 04:35:17.316783 27750 ts_tablet_manager.cc:1939] T ef70415e79c14282ab1ae2f74f1fb3ee P 0cf16ea67b6a4b66956c5dc43c0ca4fc: Deleting consensus metadata
I20260808 04:35:17.317806 26966 catalog_manager.cc:4841] TS 0cf16ea67b6a4b66956c5dc43c0ca4fc (127.15.228.66:45041): tablet ef70415e79c14282ab1ae2f74f1fb3ee (table test-table [id=6822467897e94ad082c791d12dc8ae8c]) successfully deleted
I20260808 04:35:17.319362 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 27072
I20260808 04:35:17.334108 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 27263
I20260808 04:35:17.347934 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 27443
I20260808 04:35:17.361697 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 26944
2026-08-08T04:35:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3972 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:35:17 dist-test-slave-drl0 krb5kdc[27802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:35:17 dist-test-slave-drl0 krb5kdc[27802](info): set up 2 sockets
Aug 08 04:35:17 dist-test-slave-drl0 krb5kdc[27802](info): commencing operation
krb5kdc: starting...
W20260808 04:35:18.626128 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.232s user 0.003s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:35:18 dist-test-slave-drl0 krb5kdc[27802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:35:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:35:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:35:18.771363 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:33565
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:45283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:33565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:18.898243 27822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:18.898731 27822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:18.898844 27822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:18.903117 27822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:35:18.903194 27822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:18.903230 27822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:18.903267 27822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:35:18.903301 27822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:35:18.909097 27822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:45283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:33565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:33565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:18.910234 27822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:18.911468 27822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:18.920536 27831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:18.920622 27833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:18.920603 27822 server_base.cc:1034] running on GCE node
W20260808 04:35:18.920552 27835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:18.921332 27822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:18.922461 27822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:18.923708 27822 hybrid_clock.cc:648] HybridClock initialized: now 1786163718923646 us; error 75 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T04:35:18Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1579 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:35:19 dist-test-slave-drl0 krb5kdc[27841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:35:19 dist-test-slave-drl0 krb5kdc[27841](info): set up 2 sockets
Aug 08 04:35:19 dist-test-slave-drl0 krb5kdc[27841](info): commencing operation
krb5kdc: starting...
W20260808 04:35:20.233515 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.260s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:35:20 dist-test-slave-drl0 krb5kdc[27841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:35:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:35:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:35:20.379799 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:32887
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:43263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:32887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:20.510982 27859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:20.511265 27859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:20.511310 27859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:20.515620 27859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:35:20.515690 27859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:20.515712 27859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:20.515729 27859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:35:20.515744 27859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:35:20.521265 27859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:43263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:32887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:32887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:20.522363 27859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:20.523483 27859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:20.532232 27865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:20.532320 27867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:20.532296 27864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:20.532478 27859 server_base.cc:1034] running on GCE node
I20260808 04:35:20.533236 27859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:20.534391 27859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:20.535666 27859 hybrid_clock.cc:648] HybridClock initialized: now 1786163720535593 us; error 95 us; skew 500 ppm
Aug 08 04:35:20 dist-test-slave-drl0 krb5kdc[27841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163720, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:20.539518 27859 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:35:20.540938 27859 webserver.cc:460] Webserver started at http://127.15.228.126:45503/ using document root <none> and password file <none>
I20260808 04:35:20.541599 27859 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:20.541697 27859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:20.541934 27859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:20.544154 27859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "503ac26089b54c1892c536a8c476be0b"
format_stamp: "Formatted at 2026-08-08 04:35:20 on dist-test-slave-drl0"
server_key: "e32b126aeaf48b57ed6e9d6a457a8922"
server_key_iv: "3e994b6581425d09e49a6da98f9d7c24"
server_key_version: "encryptionkey@0"
I20260808 04:35:20.544744 27859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "503ac26089b54c1892c536a8c476be0b"
format_stamp: "Formatted at 2026-08-08 04:35:20 on dist-test-slave-drl0"
server_key: "e32b126aeaf48b57ed6e9d6a457a8922"
server_key_iv: "3e994b6581425d09e49a6da98f9d7c24"
server_key_version: "encryptionkey@0"
I20260808 04:35:20.549151 27859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:35:20.552006 27880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:20.553710 27859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:35:20.553877 27859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "503ac26089b54c1892c536a8c476be0b"
format_stamp: "Formatted at 2026-08-08 04:35:20 on dist-test-slave-drl0"
server_key: "e32b126aeaf48b57ed6e9d6a457a8922"
server_key_iv: "3e994b6581425d09e49a6da98f9d7c24"
server_key_version: "encryptionkey@0"
I20260808 04:35:20.553998 27859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:20.598347 27859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:20.602080 27859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:20.602331 27859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:20.610234 27859 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:32887
I20260808 04:35:20.610323 27952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:32887 every 8 connection(s)
I20260808 04:35:20.611313 27859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:35:20.614348 27953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:20.619220 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 27859
I20260808 04:35:20.619380 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:35:20.619657 16273 external_mini_cluster.cc:1428] Setting key c9013840c0dea17dc744b7406f50a308
I20260808 04:35:20.620117 27953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b: Bootstrap starting.
I20260808 04:35:20.623597 27953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:20.624987 27953 log.cc:824] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:20.627653 27953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b: No bootstrap required, opened a new log
Aug 08 04:35:20 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:35:20.632789 27953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ac26089b54c1892c536a8c476be0b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 32887 } }
I20260808 04:35:20.633378 27953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:20.633477 27953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 503ac26089b54c1892c536a8c476be0b, State: Initialized, Role: FOLLOWER
I20260808 04:35:20.634400 27953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ac26089b54c1892c536a8c476be0b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 32887 } }
I20260808 04:35:20.634624 27953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:35:20.634692 27953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:35:20.634809 27953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:20.636103 27953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ac26089b54c1892c536a8c476be0b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 32887 } }
I20260808 04:35:20.636511 27953 leader_election.cc:302] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 503ac26089b54c1892c536a8c476be0b; no voters:
I20260808 04:35:20.636940 27953 leader_election.cc:288] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:35:20.637086 27964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:35:20.637411 27964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [term 1 LEADER]: Becoming Leader. State: Replica: 503ac26089b54c1892c536a8c476be0b, State: Running, Role: LEADER
I20260808 04:35:20.637904 27964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ac26089b54c1892c536a8c476be0b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 32887 } }
I20260808 04:35:20.638921 27959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:20.621240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37952 (local address 127.15.228.126:32887)
0808 04:35:20.621803 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:20.621820 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:20.621857 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:35:20.622815 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:20.622851 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:20.623441 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:20.623759 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:20.625076 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:20.626344 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:20.627718 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:20.628397 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:20.628700 (+ 303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:20.631840 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:20.631847 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:20.631865 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:20.631908 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:20.634713 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:20.635266 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:20.635272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:20.635279 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:20.635420 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:20.635812 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:20.635815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:20.635817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:20.636416 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:20.636663 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:20.637051 (+ 388us) server_negotiation.cc:299] Negotiation successful
0808 04:35:20.637392 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":197,"threads_started":1}
I20260808 04:35:20.639022 27953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:35:20.639679 27965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "503ac26089b54c1892c536a8c476be0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ac26089b54c1892c536a8c476be0b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 32887 } } }
I20260808 04:35:20.639833 27965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:20.640264 27968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:35:20.641680 27966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 503ac26089b54c1892c536a8c476be0b. Latest consensus state: current_term: 1 leader_uuid: "503ac26089b54c1892c536a8c476be0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ac26089b54c1892c536a8c476be0b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 32887 } } }
I20260808 04:35:20.641776 27966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:20.644143 27968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:35:20.651432 27968 catalog_manager.cc:1347] Generated new cluster ID: f7eb6ba1b1a440c29e8a16aea9fab434
I20260808 04:35:20.651558 27968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:35:20.662604 27968 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:35:20.663774 27968 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:35:20.669304 27968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b: Generated new TSK 0
I20260808 04:35:20.670019 27968 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
I20260808 04:35:20.711330 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:32887
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:43263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:20.856091 27993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:20.856498 27993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:20.856554 27993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:20.862718 27993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:20.862838 27993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:20.862920 27993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:35:20.869395 27993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:43263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:32887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:20.870579 27993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:20.871850 27993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:20.882009 28001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:20.882352 28002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:20.882490 28006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:20.882530 27993 server_base.cc:1034] running on GCE node
I20260808 04:35:20.883378 27993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:20.884224 27993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:20.885447 27993 hybrid_clock.cc:648] HybridClock initialized: now 1786163720885420 us; error 35 us; skew 500 ppm
Aug 08 04:35:20 dist-test-slave-drl0 krb5kdc[27841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163720, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:20.889065 27993 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:35:20.890120 27993 webserver.cc:460] Webserver started at http://127.15.228.65:36849/ using document root <none> and password file <none>
I20260808 04:35:20.890688 27993 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:20.890758 27993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:20.890997 27993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:20.893074 27993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "a23c55604324456591dea0e0e4a73fd5"
format_stamp: "Formatted at 2026-08-08 04:35:20 on dist-test-slave-drl0"
server_key: "667d05ef793d3bc5db5cbca36ada7460"
server_key_iv: "8312f2eca00a41f6f5549058cf4a1e78"
server_key_version: "encryptionkey@0"
I20260808 04:35:20.893642 27993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "a23c55604324456591dea0e0e4a73fd5"
format_stamp: "Formatted at 2026-08-08 04:35:20 on dist-test-slave-drl0"
server_key: "667d05ef793d3bc5db5cbca36ada7460"
server_key_iv: "8312f2eca00a41f6f5549058cf4a1e78"
server_key_version: "encryptionkey@0"
I20260808 04:35:20.897512 27993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 04:35:20.900758 28019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:20.902161 27993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:35:20.902335 27993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "a23c55604324456591dea0e0e4a73fd5"
format_stamp: "Formatted at 2026-08-08 04:35:20 on dist-test-slave-drl0"
server_key: "667d05ef793d3bc5db5cbca36ada7460"
server_key_iv: "8312f2eca00a41f6f5549058cf4a1e78"
server_key_version: "encryptionkey@0"
I20260808 04:35:20.902460 27993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:20.929765 27993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:20.934832 27993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:20.935187 27993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:20.936278 27993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:20.938004 27993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:20.938076 27993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:20.938185 27993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:20.938223 27993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:20.950088 27993 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:38073
I20260808 04:35:20.950218 28170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:38073 every 8 connection(s)
I20260808 04:35:20.951205 27993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
Aug 08 04:35:20 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163720, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:35:20.961584 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 27993
I20260808 04:35:20.961800 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:35:20.962204 16273 external_mini_cluster.cc:1428] Setting key 4c572fc5531711eff176968940f05e4a
I20260808 04:35:20.965909 27959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:20.953413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:53829 (local address 127.15.228.126:32887)
0808 04:35:20.953613 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:20.953617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:20.954317 (+ 700us) server_negotiation.cc:407] Connection header received
0808 04:35:20.955754 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:20.955757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:20.955817 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:20.955928 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:20.957101 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:20.957723 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:20.958993 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:20.959228 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:20.959345 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:20.962038 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:20.962042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:20.962045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:20.962076 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:20.963892 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:20.964404 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:20.964409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:20.964410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:20.964491 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:20.964834 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:20.964839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:20.964841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:20.965020 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:20.965129 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:20.965619 (+ 490us) server_negotiation.cc:299] Negotiation successful
0808 04:35:20.965741 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260808 04:35:20.967018 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:20.953653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.65:53829)
0808 04:35:20.954155 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:20.954198 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:20.955534 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:20.956100 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:20.956106 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:20.956473 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:20.956946 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:20.956959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:20.957862 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:20.957865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:20.958821 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:20.958831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:20.959389 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:20.959394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:20.959572 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:20.960166 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:20.960182 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:20.961828 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:20.964008 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:20.964014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:20.964029 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:20.964290 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:20.964591 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:20.964594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:20.964596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:20.964736 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:20.965129 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:20.965137 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:20.965387 (+ 250us) client_negotiation.cc:769] Sending connection context
0808 04:35:20.965579 (+ 192us) client_negotiation.cc:240] Negotiation successful
0808 04:35:20.965919 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":130,"threads_started":1}
I20260808 04:35:20.968233 28171 heartbeater.cc:347] Connected to a master server at 127.15.228.126:32887
I20260808 04:35:20.968469 28171 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:20.968989 28171 heartbeater.cc:511] Master 127.15.228.126:32887 requested a full tablet report, sending...
I20260808 04:35:20.970624 27903 ts_manager.cc:194] Registered new tserver with Master: a23c55604324456591dea0e0e4a73fd5 (127.15.228.65:38073)
I20260808 04:35:20.971926 27903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:53829
WARNING: no policy specified for kudu/127.15.228.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
I20260808 04:35:20.995255 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:32887
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:43263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:21.124557 28178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:21.124828 28178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:21.124873 28178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:21.129333 28178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:21.129438 28178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:21.129532 28178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:35:21.135032 28178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:43263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:32887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:21.136286 28178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:21.137403 28178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:21.147859 28184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:21.147852 28188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:21.147908 28186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:21.148435 28178 server_base.cc:1034] running on GCE node
I20260808 04:35:21.148844 28178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:21.149530 28178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:21.150822 28178 hybrid_clock.cc:648] HybridClock initialized: now 1786163721150765 us; error 48 us; skew 500 ppm
Aug 08 04:35:21 dist-test-slave-drl0 krb5kdc[27841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163721, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:21.155071 28178 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:35:21.156553 28178 webserver.cc:460] Webserver started at http://127.15.228.66:44133/ using document root <none> and password file <none>
I20260808 04:35:21.157142 28178 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:21.157207 28178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:21.157652 28178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:21.160496 28178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "a0f302873b3743519eb9be9b9e9880bd"
format_stamp: "Formatted at 2026-08-08 04:35:21 on dist-test-slave-drl0"
server_key: "1e3ca33a9f26eb2ef3df92b62b5746d8"
server_key_iv: "db873e380c307e6cf5b97167192edb9e"
server_key_version: "encryptionkey@0"
I20260808 04:35:21.161384 28178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "a0f302873b3743519eb9be9b9e9880bd"
format_stamp: "Formatted at 2026-08-08 04:35:21 on dist-test-slave-drl0"
server_key: "1e3ca33a9f26eb2ef3df92b62b5746d8"
server_key_iv: "db873e380c307e6cf5b97167192edb9e"
server_key_version: "encryptionkey@0"
I20260808 04:35:21.166597 28178 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:35:21.170012 28197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:21.172091 28178 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260808 04:35:21.172259 28178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "a0f302873b3743519eb9be9b9e9880bd"
format_stamp: "Formatted at 2026-08-08 04:35:21 on dist-test-slave-drl0"
server_key: "1e3ca33a9f26eb2ef3df92b62b5746d8"
server_key_iv: "db873e380c307e6cf5b97167192edb9e"
server_key_version: "encryptionkey@0"
I20260808 04:35:21.172510 28178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:21.191802 28178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:21.195329 28178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:21.195549 28178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:21.196198 28178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:21.197685 28178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:21.197772 28178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:21.197875 28178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:21.197919 28178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:21.209168 28178 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:36345
I20260808 04:35:21.209435 28341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:36345 every 8 connection(s)
I20260808 04:35:21.210199 28178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:35:21.215108 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 28178
I20260808 04:35:21.215235 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:35:21.215829 16273 external_mini_cluster.cc:1428] Setting key 34168910b50cc104d9f5b89c017d6cf2
Aug 08 04:35:21 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163721, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.228.67@KRBTEST.COM; defaulting to no policy
I20260808 04:35:21.227552 27959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:21.212326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:48359 (local address 127.15.228.126:32887)
0808 04:35:21.212648 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:21.212655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:21.213544 (+ 889us) server_negotiation.cc:407] Connection header received
0808 04:35:21.215738 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:21.215741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:21.215792 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:21.215901 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:21.217891 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:21.218474 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:21.220409 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:21.220674 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:21.220789 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:21.223500 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:21.223504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:21.223506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:21.223529 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:21.225308 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:21.225865 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:21.225868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:21.225870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:21.225954 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:21.226324 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:21.226327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:21.226329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:21.226525 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:21.226640 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:21.227234 (+ 594us) server_negotiation.cc:299] Negotiation successful
0808 04:35:21.227371 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260808 04:35:21.228420 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:21.212586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.66:48359)
0808 04:35:21.213287 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:21.213363 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:21.215266 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:21.216174 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:21.216185 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:21.216801 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:21.217656 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:21.217678 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:21.218642 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:21.218649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:21.220239 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:21.220257 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:21.220847 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:21.220851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:21.221036 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:21.221700 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:21.221717 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:21.223309 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:21.225447 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:21.225453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:21.225470 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:21.225745 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:21.226050 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:21.226054 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:21.226056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:21.226196 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:21.226639 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:21.226648 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:21.226909 (+ 261us) client_negotiation.cc:769] Sending connection context
0808 04:35:21.227174 (+ 265us) client_negotiation.cc:240] Negotiation successful
0808 04:35:21.227519 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":120,"threads_started":1}
I20260808 04:35:21.229799 28342 heartbeater.cc:347] Connected to a master server at 127.15.228.126:32887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.67@KRBTEST.COM" created.
I20260808 04:35:21.230072 28342 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:21.230593 28342 heartbeater.cc:511] Master 127.15.228.126:32887 requested a full tablet report, sending...
I20260808 04:35:21.231791 27903 ts_manager.cc:194] Registered new tserver with Master: a0f302873b3743519eb9be9b9e9880bd (127.15.228.66:36345)
I20260808 04:35:21.232388 27903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:48359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
I20260808 04:35:21.250085 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:32887
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:43263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:21.378854 28348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:21.379189 28348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:21.379266 28348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:21.383389 28348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:21.383478 28348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:21.383579 28348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:35:21.389484 28348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:43263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:32887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:21.390720 28348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:21.391983 28348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:21.402724 28362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:21.402770 28358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:21.402761 28359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:21.403337 28348 server_base.cc:1034] running on GCE node
I20260808 04:35:21.403769 28348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:21.404445 28348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:21.405633 28348 hybrid_clock.cc:648] HybridClock initialized: now 1786163721405603 us; error 50 us; skew 500 ppm
Aug 08 04:35:21 dist-test-slave-drl0 krb5kdc[27841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163721, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:21.408967 28348 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:35:21.409862 28348 webserver.cc:460] Webserver started at http://127.15.228.67:46787/ using document root <none> and password file <none>
I20260808 04:35:21.410404 28348 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:21.410463 28348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:21.410631 28348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:21.413633 28348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "323a53395f944e9b80337e1358f07be8"
format_stamp: "Formatted at 2026-08-08 04:35:21 on dist-test-slave-drl0"
server_key: "3b1f0afefeac006dc702cd7c0f88f763"
server_key_iv: "dbd64de379c74ae6bc602cf016da1b98"
server_key_version: "encryptionkey@0"
I20260808 04:35:21.414752 28348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "323a53395f944e9b80337e1358f07be8"
format_stamp: "Formatted at 2026-08-08 04:35:21 on dist-test-slave-drl0"
server_key: "3b1f0afefeac006dc702cd7c0f88f763"
server_key_iv: "dbd64de379c74ae6bc602cf016da1b98"
server_key_version: "encryptionkey@0"
I20260808 04:35:21.419625 28348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:35:21.423220 28376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:21.424934 28348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:35:21.425057 28348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "323a53395f944e9b80337e1358f07be8"
format_stamp: "Formatted at 2026-08-08 04:35:21 on dist-test-slave-drl0"
server_key: "3b1f0afefeac006dc702cd7c0f88f763"
server_key_iv: "dbd64de379c74ae6bc602cf016da1b98"
server_key_version: "encryptionkey@0"
I20260808 04:35:21.425231 28348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:21.451329 28348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:21.455577 28348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:21.455845 28348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:21.456564 28348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:21.457618 28348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:21.457677 28348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:21.457726 28348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:21.457785 28348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:21.471051 28528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:45517 every 8 connection(s)
I20260808 04:35:21.471087 28348 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:45517
I20260808 04:35:21.472527 28348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:35:21.479242 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 28348
I20260808 04:35:21.479417 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:35:21.479744 16273 external_mini_cluster.cc:1428] Setting key 113520d4d4862a47ed28e75625a2dd49
Aug 08 04:35:21 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163721, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:35:21.490262 27959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:21.475120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:43385 (local address 127.15.228.126:32887)
0808 04:35:21.475312 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:21.475317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:21.476092 (+ 775us) server_negotiation.cc:407] Connection header received
0808 04:35:21.478017 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:21.478020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:21.478069 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:21.478143 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:21.479793 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:21.480479 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:21.482500 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:21.482818 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:21.482975 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:21.486114 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:21.486117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:21.486119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:21.486144 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:21.487798 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:21.488443 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:21.488446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:21.488447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:21.488528 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:21.488904 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:21.488907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:21.488908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:21.489100 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:21.489179 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:21.489915 (+ 736us) server_negotiation.cc:299] Negotiation successful
0808 04:35:21.490079 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 04:35:21.490980 28531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:21.475405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.67:43385)
0808 04:35:21.475924 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:21.475968 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:21.477588 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:21.478346 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:21.478354 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:21.478911 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:21.479617 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:21.479638 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:21.480743 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:21.480748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:21.482363 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:21.482377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:21.483096 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:21.483100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:21.483300 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:21.484130 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:21.484150 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:21.485912 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:21.487957 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:21.487963 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:21.487978 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:21.488290 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:21.488623 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:21.488626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:21.488627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:21.488781 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:21.489208 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:21.489214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:21.489502 (+ 288us) client_negotiation.cc:769] Sending connection context
0808 04:35:21.489683 (+ 181us) client_negotiation.cc:240] Negotiation successful
0808 04:35:21.489973 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":122,"threads_started":1}
I20260808 04:35:21.492386 28529 heartbeater.cc:347] Connected to a master server at 127.15.228.126:32887
I20260808 04:35:21.492667 28529 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:21.493211 28529 heartbeater.cc:511] Master 127.15.228.126:32887 requested a full tablet report, sending...
I20260808 04:35:21.494872 27903 ts_manager.cc:194] Registered new tserver with Master: 323a53395f944e9b80337e1358f07be8 (127.15.228.67:45517)
I20260808 04:35:21.495517 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:35:21.495607 27903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:43385
WARNING: no policy specified for oryx/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.65.keytab.
Entry for principal oryx/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.65.keytab.
WARNING: no policy specified for oryx/127.15.228.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.228.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.66.keytab.
Entry for principal oryx/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.66.keytab.
WARNING: no policy specified for oryx/127.15.228.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.228.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.67.keytab.
Entry for principal oryx/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.67.keytab.
I20260808 04:35:21.593918 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 27993
I20260808 04:35:21.605651 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 28178
I20260808 04:35:21.618757 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 28348
I20260808 04:35:21.630770 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 27859
I20260808 04:35:21.646062 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:32887
--webserver_interface=127.15.228.126
--webserver_port=45503
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:43263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:32887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:21.769773 28540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:21.770195 28540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:21.770331 28540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:21.774499 28540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:35:21.774612 28540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:21.774655 28540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:21.774695 28540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:35:21.774732 28540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:35:21.780104 28540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:43263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:32887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:32887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=45503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:21.781239 28540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:21.782764 28540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:21.791877 28555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:21.791909 28549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:21.791898 28547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:21.792165 28540 server_base.cc:1034] running on GCE node
I20260808 04:35:21.792876 28540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:21.794034 28540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:21.795337 28540 hybrid_clock.cc:648] HybridClock initialized: now 1786163721795288 us; error 38 us; skew 500 ppm
Aug 08 04:35:21 dist-test-slave-drl0 krb5kdc[27841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163721, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:21.798475 28540 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:35:21.799430 28540 webserver.cc:460] Webserver started at http://127.15.228.126:45503/ using document root <none> and password file <none>
I20260808 04:35:21.800001 28540 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:21.800174 28540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:21.804873 28540 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:35:21.807722 28565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:21.809459 28540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:35:21.809631 28540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "503ac26089b54c1892c536a8c476be0b"
format_stamp: "Formatted at 2026-08-08 04:35:20 on dist-test-slave-drl0"
server_key: "e32b126aeaf48b57ed6e9d6a457a8922"
server_key_iv: "3e994b6581425d09e49a6da98f9d7c24"
server_key_version: "encryptionkey@0"
I20260808 04:35:21.810231 28540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:21.820073 28540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:21.823501 28540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:21.823707 28540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:21.831346 28540 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:32887
I20260808 04:35:21.831423 28634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:32887 every 8 connection(s)
I20260808 04:35:21.832556 28540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:35:21.834740 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 28540
I20260808 04:35:21.835294 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:38073
--local_ip_for_outbound_sockets=127.15.228.65
--tserver_master_addrs=127.15.228.126:32887
--webserver_port=36849
--webserver_interface=127.15.228.65
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:43263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.228.65
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:35:21.837836 28635 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:35:21.840517 28635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b: Bootstrap starting.
I20260808 04:35:21.861560 28635 log.cc:824] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:21.871047 28635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:35:21.871585 28635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b: Bootstrap complete.
I20260808 04:35:21.876708 28635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ac26089b54c1892c536a8c476be0b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 32887 } }
I20260808 04:35:21.877137 28635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 503ac26089b54c1892c536a8c476be0b, State: Initialized, Role: FOLLOWER
I20260808 04:35:21.877689 28635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ac26089b54c1892c536a8c476be0b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 32887 } }
I20260808 04:35:21.877808 28635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:35:21.877902 28635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:35:21.878000 28635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:35:21.881330 28635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ac26089b54c1892c536a8c476be0b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 32887 } }
I20260808 04:35:21.881868 28635 leader_election.cc:302] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 503ac26089b54c1892c536a8c476be0b; no voters:
I20260808 04:35:21.882364 28635 leader_election.cc:288] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 04:35:21.882496 28640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:35:21.882738 28640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [term 2 LEADER]: Becoming Leader. State: Replica: 503ac26089b54c1892c536a8c476be0b, State: Running, Role: LEADER
I20260808 04:35:21.883152 28640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ac26089b54c1892c536a8c476be0b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 32887 } }
I20260808 04:35:21.884047 28635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:35:21.885262 28641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "503ac26089b54c1892c536a8c476be0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ac26089b54c1892c536a8c476be0b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 32887 } } }
I20260808 04:35:21.885447 28641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:21.885205 28642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 503ac26089b54c1892c536a8c476be0b. Latest consensus state: current_term: 2 leader_uuid: "503ac26089b54c1892c536a8c476be0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ac26089b54c1892c536a8c476be0b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 32887 } } }
I20260808 04:35:21.885578 28642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:21.886231 28650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:35:21.889823 28650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:35:21.890542 28650 catalog_manager.cc:1259] Loaded cluster ID: f7eb6ba1b1a440c29e8a16aea9fab434
I20260808 04:35:21.890592 28650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:35:21.892457 28650 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:35:21.893059 28650 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 503ac26089b54c1892c536a8c476be0b: Loaded TSK: 0
I20260808 04:35:21.894024 28650 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:35:21.971127 28637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:21.971405 28637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:21.971449 28637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:21.975664 28637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:21.975729 28637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:21.975852 28637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:35:21.981691 28637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:43263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.65.keytab
--principal=oryx/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:38073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=36849
--tserver_master_addrs=127.15.228.126:32887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:21.982973 28637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:21.984081 28637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:21.997933 28672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:21.997911 28668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:21.997901 28669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:21.998435 28637 server_base.cc:1034] running on GCE node
I20260808 04:35:21.999013 28637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:22.000181 28637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:22.001495 28637 hybrid_clock.cc:648] HybridClock initialized: now 1786163722001450 us; error 64 us; skew 500 ppm
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163722, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:22.005508 28637 init.cc:377] Logged in from keytab as oryx/127.15.228.65@KRBTEST.COM (short username oryx)
I20260808 04:35:22.006597 28637 webserver.cc:460] Webserver started at http://127.15.228.65:36849/ using document root <none> and password file <none>
I20260808 04:35:22.007256 28637 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:22.007375 28637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:22.011615 28637 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:35:22.013921 28680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:22.015445 28637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:35:22.015548 28637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "a23c55604324456591dea0e0e4a73fd5"
format_stamp: "Formatted at 2026-08-08 04:35:20 on dist-test-slave-drl0"
server_key: "667d05ef793d3bc5db5cbca36ada7460"
server_key_iv: "8312f2eca00a41f6f5549058cf4a1e78"
server_key_version: "encryptionkey@0"
I20260808 04:35:22.016147 28637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:22.039160 28637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:22.043537 28637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:22.043881 28637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:22.044829 28637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:22.046273 28637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:22.046387 28637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:22.046490 28637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:22.046540 28637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:22.060091 28637 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:38073
I20260808 04:35:22.060138 28836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:38073 every 8 connection(s)
I20260808 04:35:22.061636 28637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:35:22.064513 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 28637
I20260808 04:35:22.065047 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:36345
--local_ip_for_outbound_sockets=127.15.228.66
--tserver_master_addrs=127.15.228.126:32887
--webserver_port=44133
--webserver_interface=127.15.228.66
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:43263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.228.66
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:22.076076 28841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:22.064037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.65:54829)
0808 04:35:22.064523 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:22.064575 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:22.066148 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:22.067450 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:22.067458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:22.067894 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:22.068376 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:22.068391 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.069663 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.069671 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:22.070609 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:22.070620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.071602 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.071607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:22.071792 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:22.072434 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:22.072452 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:22.075084 (+ 2632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":154,"threads_started":1}
W20260808 04:35:22.076066 28843 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:22.064004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:54829 (local address 127.15.228.126:32887)
0808 04:35:22.064622 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:22.064644 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:22.064789 (+ 145us) server_negotiation.cc:407] Connection header received
0808 04:35:22.066572 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:22.066589 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:22.067049 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:22.067371 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:22.068555 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.069515 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.070741 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.071395 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.071611 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:22.075296 (+ 3685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.65:54829: BlockingRecv error: recv got EOF from 127.15.228.65:54829 (error 108)
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":155,"threads_started":1}
W20260808 04:35:22.076746 28837 heartbeater.cc:644] Failed to heartbeat to 127.15.228.126:32887 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.228.126:32887: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:22.083630 28841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:22.077213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.65:47269)
0808 04:35:22.077403 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:22.077419 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:22.077507 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:22.077836 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:22.077839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:22.078015 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:22.078128 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:22.078134 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.078896 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.078900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:22.079849 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:22.079861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.080423 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.080427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:22.080521 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:22.081052 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:22.081067 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:22.083477 (+ 2410us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88}
W20260808 04:35:22.084105 28843 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:22.077326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:47269 (local address 127.15.228.126:32887)
0808 04:35:22.077442 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:22.077445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:22.077461 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:35:22.077659 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:22.077661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:22.077713 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:22.077793 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:22.078284 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.078791 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.079991 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.080306 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.080401 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:22.083886 (+ 3485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.65:47269: BlockingRecv error: recv got EOF from 127.15.228.65:47269 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:22.091425 28841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:22.084779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.65:46415)
0808 04:35:22.084934 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:22.084951 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:22.085039 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:22.085345 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:22.085348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:22.085516 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:22.085611 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:22.085617 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.086796 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.086799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:22.087410 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:22.087420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.088082 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.088088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:22.088194 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:22.088896 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:22.088910 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:22.091283 (+ 2373us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260808 04:35:22.091668 28837 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:35:22.091794 28843 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:22.084836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:46415 (local address 127.15.228.126:32887)
0808 04:35:22.084984 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:22.084988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:22.085005 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:35:22.085162 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:22.085164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:22.085220 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:22.085299 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:22.085835 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.086659 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.087528 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.087889 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.087999 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:22.091457 (+ 3458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.65:46415: BlockingRecv error: recv got EOF from 127.15.228.65:46415 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260808 04:35:22.192528 28844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:22.192803 28844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:22.192852 28844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:22.196874 28844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:22.196947 28844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:22.197010 28844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:35:22.202430 28844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:43263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.66.keytab
--principal=oryx/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:36345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=44133
--tserver_master_addrs=127.15.228.126:32887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:22.203586 28844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:22.204622 28844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:22.215651 28854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:22.215742 28851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:22.215746 28850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:22.215973 28844 server_base.cc:1034] running on GCE node
I20260808 04:35:22.216471 28844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:22.217142 28844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:22.218397 28844 hybrid_clock.cc:648] HybridClock initialized: now 1786163722218371 us; error 43 us; skew 500 ppm
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163722, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:22.221864 28844 init.cc:377] Logged in from keytab as oryx/127.15.228.66@KRBTEST.COM (short username oryx)
I20260808 04:35:22.223335 28844 webserver.cc:460] Webserver started at http://127.15.228.66:44133/ using document root <none> and password file <none>
I20260808 04:35:22.224066 28844 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:22.224200 28844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:22.229280 28844 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:35:22.232049 28862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:22.233484 28844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:35:22.233597 28844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "a0f302873b3743519eb9be9b9e9880bd"
format_stamp: "Formatted at 2026-08-08 04:35:21 on dist-test-slave-drl0"
server_key: "1e3ca33a9f26eb2ef3df92b62b5746d8"
server_key_iv: "db873e380c307e6cf5b97167192edb9e"
server_key_version: "encryptionkey@0"
I20260808 04:35:22.234247 28844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:22.247869 28844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:22.252003 28844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:22.252341 28844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:22.253063 28844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:22.254088 28844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:22.254151 28844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:22.254196 28844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:22.254263 28844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:22.265031 28844 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:36345
I20260808 04:35:22.265054 29020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:36345 every 8 connection(s)
I20260808 04:35:22.266181 28844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:35:22.274262 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 28844
I20260808 04:35:22.274812 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:45517
--local_ip_for_outbound_sockets=127.15.228.67
--tserver_master_addrs=127.15.228.126:32887
--webserver_port=46787
--webserver_interface=127.15.228.67
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:43263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.228.67
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 04:35:22.281438 28843 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:22.268288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:56017 (local address 127.15.228.126:32887)
0808 04:35:22.268472 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:22.268477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:22.269730 (+ 1253us) server_negotiation.cc:407] Connection header received
0808 04:35:22.271613 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:22.271617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:22.271700 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:22.271815 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:22.273144 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.273793 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.275741 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.276027 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.276160 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:22.281245 (+ 5085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.66:56017: BlockingRecv error: recv got EOF from 127.15.228.66:56017 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 04:35:22.281840 29023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:22.268659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.66:56017)
0808 04:35:22.269448 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:22.269528 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:22.271275 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:22.271992 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:22.271999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:22.272405 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:22.272936 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:22.272950 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.274036 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.274041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:22.275514 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:22.275530 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.276242 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.276249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:22.276493 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:22.277474 (+ 981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:22.277500 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:22.280910 (+ 3410us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":125,"threads_started":1}
W20260808 04:35:22.282510 29021 heartbeater.cc:644] Failed to heartbeat to 127.15.228.126:32887 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.228.126:32887: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:22.289728 29023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:22.283310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.66:49801)
0808 04:35:22.283454 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:22.283469 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:22.283562 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:22.283878 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:22.283881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:22.284044 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:22.284223 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:22.284230 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.285424 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.285427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:22.285985 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:22.285993 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.286837 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.286842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:22.287087 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:22.287802 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:22.287817 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:22.289626 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 04:35:22.289932 28843 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:22.283398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:49801 (local address 127.15.228.126:32887)
0808 04:35:22.283554 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:22.283561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:22.283581 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:35:22.283642 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:22.283648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:22.283715 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:22.283825 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:22.284357 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.285295 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.286132 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.286563 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.286703 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:22.289799 (+ 3096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.66:49801: BlockingRecv error: recv got EOF from 127.15.228.66:49801 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:22.296602 29023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:22.290467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.66:54433)
0808 04:35:22.290599 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:22.290613 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:22.290691 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:22.291069 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:22.291072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:22.291235 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:22.291327 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:22.291333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.292370 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.292375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:22.293252 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:22.293262 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.293767 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.293770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:22.293865 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:22.294321 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:22.294336 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:22.296450 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 04:35:22.296692 28843 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:22.290567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:54433 (local address 127.15.228.126:32887)
0808 04:35:22.290677 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:22.290680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:22.290692 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:35:22.290842 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:22.290845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:22.290887 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:22.291052 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:22.291465 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.292190 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.293404 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.293648 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.293745 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:22.296573 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.66:54433: BlockingRecv error: recv got EOF from 127.15.228.66:54433 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260808 04:35:22.296921 29021 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:35:22.407267 29024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:22.407567 29024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:22.407637 29024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:22.413704 29024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:22.413832 29024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:22.413938 29024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:35:22.421527 29024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:43263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/krb5kdc/oryx_127.15.228.67.keytab
--principal=oryx/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:45517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=46787
--tserver_master_addrs=127.15.228.126:32887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:22.422771 29024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:22.424044 29024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:22.434438 29033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:22.434634 29034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:22.434649 29036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:22.434841 29024 server_base.cc:1034] running on GCE node
I20260808 04:35:22.435297 29024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:22.436005 29024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:22.437288 29024 hybrid_clock.cc:648] HybridClock initialized: now 1786163722437259 us; error 49 us; skew 500 ppm
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163722, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:22.441493 29024 init.cc:377] Logged in from keytab as oryx/127.15.228.67@KRBTEST.COM (short username oryx)
I20260808 04:35:22.442786 29024 webserver.cc:460] Webserver started at http://127.15.228.67:46787/ using document root <none> and password file <none>
I20260808 04:35:22.443495 29024 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:22.443624 29024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:22.448305 29024 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:35:22.450618 29047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:22.452178 29024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:35:22.452358 29024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "323a53395f944e9b80337e1358f07be8"
format_stamp: "Formatted at 2026-08-08 04:35:21 on dist-test-slave-drl0"
server_key: "3b1f0afefeac006dc702cd7c0f88f763"
server_key_iv: "dbd64de379c74ae6bc602cf016da1b98"
server_key_version: "encryptionkey@0"
I20260808 04:35:22.452996 29024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:22.479403 29024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:22.483560 29024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:22.483785 29024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:22.484462 29024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:22.485420 29024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:22.485493 29024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:22.485569 29024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:22.485623 29024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:22.497364 29024 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:45517
I20260808 04:35:22.497416 29205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:45517 every 8 connection(s)
I20260808 04:35:22.498502 29024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:35:22.504962 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 29024
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 04:35:22.512526 28843 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:22.500646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:39397 (local address 127.15.228.126:32887)
0808 04:35:22.500836 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:22.500841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:22.501709 (+ 868us) server_negotiation.cc:407] Connection header received
0808 04:35:22.503076 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:22.503079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:22.503130 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:22.503313 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:22.504599 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.505227 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.507290 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.507653 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.507798 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:22.512340 (+ 4542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.67:39397: BlockingRecv error: recv got EOF from 127.15.228.67:39397 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 04:35:22.513451 29210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:22.501021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.67:39397)
0808 04:35:22.501538 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:22.501589 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:22.502854 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:22.503450 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:22.503457 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:22.503827 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:22.504325 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:22.504338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.505479 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.505486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:22.507088 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:22.507109 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.507904 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.507909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:22.508092 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:22.508910 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:22.508932 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:22.512128 (+ 3196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":118,"threads_started":1}
W20260808 04:35:22.514276 29206 heartbeater.cc:644] Failed to heartbeat to 127.15.228.126:32887 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.228.126:32887: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
I20260808 04:35:22.518952 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:22.506749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37954 (local address 127.15.228.126:32887)
0808 04:35:22.507142 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:22.507150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:22.507167 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:35:22.507233 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:22.507236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:22.507299 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:22.507390 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:22.508465 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.509317 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.510447 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.510764 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.510887 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:22.513018 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:22.513024 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:22.513041 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:22.513068 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:22.515840 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:22.516393 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:22.516399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:22.516405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:22.516555 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:22.517138 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:22.517145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:22.517148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:22.517620 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:22.517887 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:22.518395 (+ 508us) server_negotiation.cc:299] Negotiation successful
0808 04:35:22.518686 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":197,"threads_started":1}
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:22.520911 29210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:22.514786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.67:42517)
0808 04:35:22.515004 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:22.515021 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:22.515103 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:22.515429 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:22.515432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:22.515606 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:22.515710 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:22.515716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.516649 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.516653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:22.517304 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:22.517313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.517765 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.517769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:22.517869 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:22.518395 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:22.518409 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:22.520789 (+ 2380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260808 04:35:22.521059 28843 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:22.514972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:42517 (local address 127.15.228.126:32887)
0808 04:35:22.515098 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:22.515102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:22.515116 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:35:22.515270 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:22.515273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:22.515320 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:22.515409 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:22.515826 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.516539 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.517428 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.517649 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.517761 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:22.520928 (+ 3167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.67:42517: BlockingRecv error: recv got EOF from 127.15.228.67:42517 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:22 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:22.527525 29210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:22.521893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.67:51989)
0808 04:35:22.522290 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:22.522305 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:22.522407 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:22.522618 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:22.522620 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:22.522741 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:22.522809 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:22.522813 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.523651 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.523656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:22.524279 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:22.524293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.524828 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.524831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:22.524910 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:22.525347 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:22.525360 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:22.527424 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304}
W20260808 04:35:22.527758 29206 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:35:22.527927 28843 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:22.521941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:51989 (local address 127.15.228.126:32887)
0808 04:35:22.522082 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:22.522085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:22.522383 (+ 298us) server_negotiation.cc:407] Connection header received
0808 04:35:22.522476 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:22.522478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:22.522513 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:22.522577 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:22.522915 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.523456 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.524459 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:22.524680 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:22.524776 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:22.527575 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.67:51989: BlockingRecv error: recv got EOF from 127.15.228.67:51989 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 08 04:35:23 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:23 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:23.100435 29214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:23.092611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.65:47643)
0808 04:35:23.093013 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:23.093030 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:23.093292 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:23.093823 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:23.093827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:23.094115 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:23.094297 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:23.094304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:23.095428 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:23.095433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:23.096177 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:23.096188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:23.096817 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:23.096821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:23.097011 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:23.097620 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:23.097633 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:23.100237 (+ 2604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":158,"threads_started":1}
W20260808 04:35:23.100492 29215 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:23.092827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:47643 (local address 127.15.228.126:32887)
0808 04:35:23.093132 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:23.093136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:23.093153 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:35:23.093485 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:23.093488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:23.093547 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:23.093663 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:23.094481 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:23.095221 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:23.096347 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:23.096681 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:23.096809 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:23.100368 (+ 3559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.65:47643: BlockingRecv error: recv got EOF from 127.15.228.65:47643 (error 108)
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":119,"threads_started":1}
Aug 08 04:35:23 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:23 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:23.304278 29216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:23.297832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.66:60111)
0808 04:35:23.298422 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:23.298443 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:23.298677 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:23.299039 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:23.299043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:23.299252 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:23.299374 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:23.299380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:23.300303 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:23.300306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:23.300897 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:23.300907 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:23.301446 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:23.301449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:23.301517 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:23.302148 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:23.302160 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:23.304146 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":334,"threads_started":1}
W20260808 04:35:23.304520 29215 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:23.297970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:60111 (local address 127.15.228.126:32887)
0808 04:35:23.298236 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:23.298240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:23.298635 (+ 395us) server_negotiation.cc:407] Connection header received
0808 04:35:23.298794 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:23.298796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:23.298853 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:23.298969 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:23.299528 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:23.300072 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:23.301067 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:23.301285 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:23.301387 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:23.304358 (+ 2971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.66:60111: BlockingRecv error: recv got EOF from 127.15.228.66:60111 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
Aug 08 04:35:23 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:23 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:23.536204 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:23.528532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.67:41335)
0808 04:35:23.529009 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:23.529023 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:23.529133 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:23.529412 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:23.529415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:23.529627 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:23.529752 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:23.529758 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:23.530684 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:23.530687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:23.531520 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:23.531532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:23.532153 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:23.532189 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:23.532302 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:23.533059 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:23.533075 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:23.536064 (+ 2989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":279,"threads_started":1}
W20260808 04:35:23.536682 29215 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:23.528711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:41335 (local address 127.15.228.126:32887)
0808 04:35:23.528863 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:23.528867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:23.529108 (+ 241us) server_negotiation.cc:407] Connection header received
0808 04:35:23.529221 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:23.529223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:23.529273 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:23.529338 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:23.529920 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:23.530529 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:23.531675 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:23.531903 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:23.531998 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:23.536455 (+ 4457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.67:41335: BlockingRecv error: recv got EOF from 127.15.228.67:41335 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 04:35:24 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:24 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:24.108755 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:24.101529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.65:35423)
0808 04:35:24.101859 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:24.101874 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:24.101973 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:24.102264 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:24.102266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:24.102470 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:24.102577 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:24.102583 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:24.103495 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:24.103498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:24.104517 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:24.104545 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:24.105429 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:24.105433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:24.105548 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:24.106291 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:24.106303 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:24.108615 (+ 2312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":131,"threads_started":1}
W20260808 04:35:24.109035 29219 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:24.101695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:35423 (local address 127.15.228.126:32887)
0808 04:35:24.101962 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:24.101966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:24.101982 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:35:24.102045 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:24.102048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:24.102092 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:24.102187 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:24.102735 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:24.103318 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:24.104822 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:24.105205 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:24.105376 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:24.108897 (+ 3521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.65:35423: BlockingRecv error: recv got EOF from 127.15.228.65:35423 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":109,"threads_started":1}
Aug 08 04:35:24 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:24 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:24.312214 29221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:24.305409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.66:46189)
0808 04:35:24.305719 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:24.305732 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:24.305844 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:24.306128 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:24.306131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:24.306317 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:24.306426 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:24.306432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:24.307538 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:24.307542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:24.308413 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:24.308422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:24.308988 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:24.308991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:24.309069 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:24.309727 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:24.309739 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:24.312056 (+ 2317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":126,"threads_started":1}
W20260808 04:35:24.312388 29219 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:24.305533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:46189 (local address 127.15.228.126:32887)
0808 04:35:24.305681 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:24.305685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:24.305840 (+ 155us) server_negotiation.cc:407] Connection header received
0808 04:35:24.305911 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:24.305913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:24.305960 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:24.306033 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:24.306572 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:24.307377 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:24.308607 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:24.308834 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:24.308928 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:24.312237 (+ 3309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.66:46189: BlockingRecv error: recv got EOF from 127.15.228.66:46189 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 08 04:35:24 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:24 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:24.544080 29222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:24.537648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.67:51571)
0808 04:35:24.538140 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:24.538160 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:24.538328 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:24.538715 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:24.538719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:24.539019 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:24.539184 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:24.539193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:24.539966 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:24.539971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:24.540533 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:24.540541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:24.541072 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:24.541075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:24.541158 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:24.541814 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:24.541826 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:24.543923 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":134,"threads_started":1}
W20260808 04:35:24.544224 29219 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:24.537850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:51571 (local address 127.15.228.126:32887)
0808 04:35:24.538034 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:24.538042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:24.538277 (+ 235us) server_negotiation.cc:407] Connection header received
0808 04:35:24.538466 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:24.538472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:24.538545 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:24.538674 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:24.539299 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:24.539854 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:24.540658 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:24.540934 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:24.541061 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:24.544092 (+ 3031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.67:51571: BlockingRecv error: recv got EOF from 127.15.228.67:51571 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 08 04:35:25 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:25 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:25.116645 29223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:25.109695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.65:38133)
0808 04:35:25.110004 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:25.110019 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:25.110109 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:25.110339 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:25.110341 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:25.110537 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:25.110636 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:25.110641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:25.111873 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:25.111877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:25.112556 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:25.112572 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:25.113241 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:25.113245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:25.113335 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:25.113993 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:25.114015 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:25.116455 (+ 2440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":117,"threads_started":1}
W20260808 04:35:25.116775 29225 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:25.109850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:38133 (local address 127.15.228.126:32887)
0808 04:35:25.110124 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:25.110127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:25.110142 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:35:25.110179 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:25.110181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:25.110222 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:25.110307 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:25.110894 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:25.111675 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:25.112762 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:25.113032 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:25.113156 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:25.116622 (+ 3466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.65:38133: BlockingRecv error: recv got EOF from 127.15.228.65:38133 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":120,"threads_started":1}
Aug 08 04:35:25 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:25 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:25.319948 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:25.313200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.66:47519)
0808 04:35:25.313621 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:25.313635 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:25.313786 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:25.314117 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:25.314119 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:25.314392 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:25.314532 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:25.314543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:25.315557 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:25.315563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:25.316126 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:25.316133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:25.316632 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:25.316635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:25.316718 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:25.317278 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:25.317292 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:25.319767 (+ 2475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":135,"threads_started":1}
W20260808 04:35:25.320019 29225 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:25.313361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:47519 (local address 127.15.228.126:32887)
0808 04:35:25.313510 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:25.313517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:25.313747 (+ 230us) server_negotiation.cc:407] Connection header received
0808 04:35:25.313909 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:25.313912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:25.313990 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:25.314075 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:25.314676 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:25.315445 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:25.316250 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:25.316511 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:25.316628 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:25.319908 (+ 3280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.66:47519: BlockingRecv error: recv got EOF from 127.15.228.66:47519 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 08 04:35:25 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:25 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:25.552820 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:25.545267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.67:37085)
0808 04:35:25.545659 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:25.545673 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:25.545773 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:25.546140 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:25.546143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:25.546353 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:25.546472 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:25.546478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:25.547512 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:25.547515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:25.548235 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:25.548246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:25.548857 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:25.548860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:25.548925 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:25.549603 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:25.549621 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:25.552660 (+ 3039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":168,"threads_started":1}
W20260808 04:35:25.553349 29225 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:25.545538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:37085 (local address 127.15.228.126:32887)
0808 04:35:25.545754 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:25.545758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:25.545779 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:35:25.545874 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:25.545876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:25.545940 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:25.546052 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:25.546683 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:25.547333 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:25.548417 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:25.548688 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:25.548801 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:25.553170 (+ 4369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.67:37085: BlockingRecv error: recv got EOF from 127.15.228.67:37085 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 08 04:35:26 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:26 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:26.124827 29229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:26.117769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.65:34509)
0808 04:35:26.118101 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:26.118115 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:26.118226 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:26.118721 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:26.118725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:26.119371 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:26.119500 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:26.119508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:26.120463 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:26.120466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:26.121128 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:26.121138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:26.121632 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:26.121635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:26.121700 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:26.122330 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:26.122342 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:26.124699 (+ 2357us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":103,"threads_started":1}
W20260808 04:35:26.125128 29230 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:26.117856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:34509 (local address 127.15.228.126:32887)
0808 04:35:26.118149 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:26.118153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:26.118170 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:35:26.118399 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:26.118402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:26.118478 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:26.118598 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:26.119662 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:26.120269 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:26.121258 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:26.121485 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:26.121585 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:26.124875 (+ 3290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.65:34509: BlockingRecv error: recv got EOF from 127.15.228.65:34509 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":73,"threads_started":1}
Aug 08 04:35:26 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:26 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:26.327180 29231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:26.320940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.66:46025)
0808 04:35:26.321244 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:26.321258 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:26.321367 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:26.321696 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:26.321699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:26.321898 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:26.322017 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:26.322023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:26.322828 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:26.322831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:26.323623 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:26.323633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:26.324095 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:26.324098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:26.324161 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:26.324681 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:26.324694 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:26.327000 (+ 2306us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":104,"threads_started":1}
W20260808 04:35:26.327236 29230 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:26.321027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:46025 (local address 127.15.228.126:32887)
0808 04:35:26.321187 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:26.321190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:26.321312 (+ 122us) server_negotiation.cc:407] Connection header received
0808 04:35:26.321505 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:26.321508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:26.321563 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:26.321641 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:26.322161 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:26.322705 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:26.323766 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:26.323988 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:26.324086 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:26.327119 (+ 3033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.66:46025: BlockingRecv error: recv got EOF from 127.15.228.66:46025 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 04:35:26 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:26 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:26.562625 29232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:26.554012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.67:35959)
0808 04:35:26.554371 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:26.554387 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:26.554631 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:26.555483 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:26.555486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:26.555781 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:26.555906 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:26.555914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:26.556911 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:26.556915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:26.557756 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:26.557770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:26.558897 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:26.558901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:26.559107 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:26.560017 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:26.560032 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:26.562486 (+ 2454us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
W20260808 04:35:26.563140 29230 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:26.554203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:35959 (local address 127.15.228.126:32887)
0808 04:35:26.554426 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:26.554430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:26.554452 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:35:26.555193 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:26.555201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:26.555295 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:26.555399 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:26.556077 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:26.556689 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:26.557970 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:26.558466 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:26.558827 (+ 361us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:26.562977 (+ 4150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.67:35959: BlockingRecv error: recv got EOF from 127.15.228.67:35959 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Aug 08 04:35:27 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:27 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:27.133216 29233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:27.125897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.65:36569)
0808 04:35:27.126191 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:27.126205 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:27.126290 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:27.126630 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:27.126633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:27.126854 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:27.126998 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:27.127005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:27.127888 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:27.127891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:27.128557 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:27.128566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:27.129457 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:27.129460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:27.129532 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:27.130115 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:27.130127 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:27.133082 (+ 2955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":100,"threads_started":1}
W20260808 04:35:27.133563 29235 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:27.125978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:36569 (local address 127.15.228.126:32887)
0808 04:35:27.126245 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:27.126248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:27.126263 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:35:27.126425 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:27.126428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:27.126484 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:27.126570 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:27.127137 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:27.127724 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:27.128832 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:27.129243 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:27.129425 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:27.133417 (+ 3992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.65:36569: BlockingRecv error: recv got EOF from 127.15.228.65:36569 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":104,"threads_started":1}
Aug 08 04:35:27 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:27 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:27.335745 29236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:27.328263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.66:50053)
0808 04:35:27.328695 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:27.328710 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:27.328817 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:27.329106 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:27.329109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:27.329312 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:27.329414 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:27.329419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:27.330322 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:27.330327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:27.331061 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:27.331078 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:27.331899 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:27.331902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:27.331981 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:27.332667 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:27.332684 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:27.335561 (+ 2877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":225,"threads_started":1}
W20260808 04:35:27.335902 29235 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:27.328457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:50053 (local address 127.15.228.126:32887)
0808 04:35:27.328712 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:27.328716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:27.328815 (+ 99us) server_negotiation.cc:407] Connection header received
0808 04:35:27.328907 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:27.328910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:27.328967 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:27.329033 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:27.329553 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:27.330104 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:27.331312 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:27.331711 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:27.331825 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:27.335762 (+ 3937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.66:50053: BlockingRecv error: recv got EOF from 127.15.228.66:50053 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
Aug 08 04:35:27 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:27 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:27.569998 29237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:27.563752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.67:39235)
0808 04:35:27.564155 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:27.564170 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:27.564336 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:27.564658 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:27.564662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:27.564924 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:27.565028 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:27.565035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:27.565910 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:27.565917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:27.566541 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:27.566550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:27.567125 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:27.567129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:27.567223 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:27.567783 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:27.567798 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:27.569810 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":135,"threads_started":1}
W20260808 04:35:27.570079 29235 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:27.563752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:39235 (local address 127.15.228.126:32887)
0808 04:35:27.564027 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:27.564034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:27.564277 (+ 243us) server_negotiation.cc:407] Connection header received
0808 04:35:27.564459 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:27.564463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:27.564527 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:27.564630 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:27.565176 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:27.565786 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:27.566675 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:27.566980 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:27.567116 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:27.569962 (+ 2846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.67:39235: BlockingRecv error: recv got EOF from 127.15.228.67:39235 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Aug 08 04:35:28 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:28 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:28.141214 29238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:28.134165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.65:46083)
0808 04:35:28.134561 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:28.134577 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:28.134709 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:28.135132 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:28.135135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:28.135331 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:28.135436 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:28.135441 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:28.136272 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:28.136275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:28.136945 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:28.136954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:28.137761 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:28.137766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:28.137861 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:28.138702 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:28.138717 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:28.141085 (+ 2368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":114,"threads_started":1}
W20260808 04:35:28.141518 29240 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:28.134341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:46083 (local address 127.15.228.126:32887)
0808 04:35:28.134601 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:28.134605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:28.134645 (+ 40us) server_negotiation.cc:407] Connection header received
0808 04:35:28.134841 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:28.134844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:28.134901 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:28.135041 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:28.135582 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:28.136141 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:28.137104 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:28.137513 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:28.137689 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:28.141333 (+ 3644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.65:46083: BlockingRecv error: recv got EOF from 127.15.228.65:46083 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":67,"threads_started":1}
Aug 08 04:35:28 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:28 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:28.343655 29241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:28.336902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.66:38325)
0808 04:35:28.337349 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:28.337367 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:28.337532 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:28.337839 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:28.337921 (+ 82us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:28.338182 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:28.338280 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:28.338286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:28.339167 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:28.339172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:28.339738 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:28.339746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:28.340220 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:28.340224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:28.340305 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:28.340913 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:28.340926 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:28.343468 (+ 2542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":155,"threads_started":1}
W20260808 04:35:28.343719 29240 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:28.336982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:38325 (local address 127.15.228.126:32887)
0808 04:35:28.337289 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:28.337294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:28.337460 (+ 166us) server_negotiation.cc:407] Connection header received
0808 04:35:28.337663 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:28.337665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:28.337720 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:28.337810 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:28.338426 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:28.339058 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:28.339872 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:28.340097 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:28.340208 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:28.343605 (+ 3397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.66:38325: BlockingRecv error: recv got EOF from 127.15.228.66:38325 (error 108)
Metrics: {"server-negotiator.queue_time_us":160}
Aug 08 04:35:28 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:28 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:28.579149 29243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:28.571171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.67:50741)
0808 04:35:28.571481 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:28.571497 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:28.571610 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:28.572127 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:28.572131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:28.572399 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:28.572612 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:28.572620 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:28.573874 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:28.573878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:28.574553 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:28.574563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:28.575175 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:28.575179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:28.575278 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:28.576048 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:28.576061 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:28.578881 (+ 2820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":111,"threads_started":1}
W20260808 04:35:28.579280 29240 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:28.571295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:50741 (local address 127.15.228.126:32887)
0808 04:35:28.571448 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:28.571452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:28.571665 (+ 213us) server_negotiation.cc:407] Connection header received
0808 04:35:28.571761 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:28.571763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:28.571849 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:28.571987 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:28.572802 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:28.573517 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:28.574714 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:28.574996 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:28.575100 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:28.579135 (+ 4035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.67:50741: BlockingRecv error: recv got EOF from 127.15.228.67:50741 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 08 04:35:29 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:29 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:29.153347 29244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:29.142503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.65:51973)
0808 04:35:29.143010 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:29.143027 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:29.143170 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:29.143568 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:29.143571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:29.143826 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:29.143943 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:29.143950 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:29.144918 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:29.144924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:29.145609 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:29.145620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:29.146114 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:29.146118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:29.146201 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:29.150414 (+ 4213us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:29.150437 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:29.153128 (+ 2691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":109,"threads_started":1}
W20260808 04:35:29.153502 29246 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:29.142696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:51973 (local address 127.15.228.126:32887)
0808 04:35:29.143199 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:29.143204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:29.143239 (+ 35us) server_negotiation.cc:407] Connection header received
0808 04:35:29.143289 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:29.143291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:29.143345 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:29.143459 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:29.144096 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:29.144700 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:29.145745 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:29.145986 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:29.146103 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:29.153339 (+ 7236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.65:51973: BlockingRecv error: recv got EOF from 127.15.228.65:51973 (error 108)
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":112,"threads_started":1}
Aug 08 04:35:29 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:29 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:29.350541 29248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:29.344570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.66:40601)
0808 04:35:29.344966 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:29.344980 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:29.345089 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:29.345454 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:29.345459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:29.345648 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:29.345863 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:29.345875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:29.346776 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:29.346779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:29.347362 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:29.347381 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:29.347876 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:29.347879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:29.347958 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:29.348552 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:29.348564 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:29.350419 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":89,"threads_started":1}
W20260808 04:35:29.351169 29246 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:29.344697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:40601 (local address 127.15.228.126:32887)
0808 04:35:29.344989 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:29.344993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:29.345048 (+ 55us) server_negotiation.cc:407] Connection header received
0808 04:35:29.345192 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:29.345195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:29.345262 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:29.345375 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:29.346066 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:29.346641 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:29.347506 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:29.347738 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:29.347830 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:29.350858 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.66:40601: BlockingRecv error: recv got EOF from 127.15.228.66:40601 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
Aug 08 04:35:29 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:29 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:29.587756 29249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:29.580384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.67:36187)
0808 04:35:29.580800 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:29.580815 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:29.580965 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:29.581695 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:29.581698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:29.582004 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:29.582113 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:29.582120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:29.583165 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:29.583168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:29.583817 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:29.583826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:29.584378 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:29.584382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:29.584476 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:29.585153 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:29.585166 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:29.587540 (+ 2374us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":100,"threads_started":1}
W20260808 04:35:29.587853 29246 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:29.580527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:36187 (local address 127.15.228.126:32887)
0808 04:35:29.580841 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:29.580848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:29.580889 (+ 41us) server_negotiation.cc:407] Connection header received
0808 04:35:29.581331 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:29.581345 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:29.581426 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:29.581529 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:29.582229 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:29.582996 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:29.583972 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:29.584241 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:29.584374 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:29.587709 (+ 3335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.67:36187: BlockingRecv error: recv got EOF from 127.15.228.67:36187 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 08 04:35:30 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:30 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:30.162776 29251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:30.154604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.65:44351)
0808 04:35:30.155179 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:30.155194 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:30.155301 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:30.155785 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:30.155790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:30.156080 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:30.156285 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:30.156292 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:30.157311 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:30.157314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:30.157929 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:30.157939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:30.158661 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:30.158666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:30.158770 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:30.159669 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:30.159685 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:30.162560 (+ 2875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":203,"threads_started":1}
W20260808 04:35:30.162976 29252 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:30.155075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:44351 (local address 127.15.228.126:32887)
0808 04:35:30.155326 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:30.155330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:30.155346 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:35:30.155386 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:30.155388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:30.155440 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:30.155524 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:30.156575 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:30.157164 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:30.158164 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:30.158410 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:30.158516 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:30.162784 (+ 4268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.65:44351: BlockingRecv error: recv got EOF from 127.15.228.65:44351 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":91,"threads_started":1}
Aug 08 04:35:30 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:30 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:30.359881 29253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:30.351479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.66:33741)
0808 04:35:30.352125 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:30.352154 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:30.352342 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:30.352716 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:30.352719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:30.352991 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:30.353249 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:30.353259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:30.354634 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:30.354652 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:30.355431 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:30.355444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:30.355990 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:30.355994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:30.356095 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:30.356912 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:30.356925 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:30.359739 (+ 2814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":298,"threads_started":1}
W20260808 04:35:30.360261 29252 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:30.351661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:33741 (local address 127.15.228.126:32887)
0808 04:35:30.351814 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:30.351818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:30.352328 (+ 510us) server_negotiation.cc:407] Connection header received
0808 04:35:30.352456 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:30.352459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:30.352562 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:30.352660 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:30.353465 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:30.354344 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:30.355586 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:30.355824 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:30.356100 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:30.360096 (+ 3996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.66:33741: BlockingRecv error: recv got EOF from 127.15.228.66:33741 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 08 04:35:30 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:30 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:30.595528 29254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:30.588986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.67:37155)
0808 04:35:30.589333 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:30.589351 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:30.589514 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:30.589872 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:30.589875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:30.590109 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:30.590209 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:30.590215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:30.591049 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:30.591055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:30.591620 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:30.591628 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:30.592188 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:30.592192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:30.592284 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:30.592928 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:30.592950 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:30.595396 (+ 2446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":98,"threads_started":1}
W20260808 04:35:30.595821 29252 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:30.589144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:37155 (local address 127.15.228.126:32887)
0808 04:35:30.589318 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:30.589323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:30.589449 (+ 126us) server_negotiation.cc:407] Connection header received
0808 04:35:30.589658 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:30.589663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:30.589725 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:30.589849 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:30.590342 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:30.590880 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:30.591747 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:30.592052 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:30.592180 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:30.595557 (+ 3377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.67:37155: BlockingRecv error: recv got EOF from 127.15.228.67:37155 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 08 04:35:31 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:31 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:31.170063 29255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:31.164006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.65:42971)
0808 04:35:31.164353 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:31.164367 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:31.164482 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:31.164799 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:31.164802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:31.165010 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:31.165117 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:31.165122 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:31.165994 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:31.165997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:31.166560 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:31.166569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:31.167163 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:31.167166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:31.167232 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:31.167852 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:31.167865 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:31.169944 (+ 2079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":112,"threads_started":1}
W20260808 04:35:31.170282 29256 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:31.164052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:42971 (local address 127.15.228.126:32887)
0808 04:35:31.164358 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:31.164361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:31.164479 (+ 118us) server_negotiation.cc:407] Connection header received
0808 04:35:31.164552 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:31.164554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:31.164604 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:31.164695 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:31.165307 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:31.165851 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:31.166717 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:31.167039 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:31.167134 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:31.170118 (+ 2984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.65:42971: BlockingRecv error: recv got EOF from 127.15.228.65:42971 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
Aug 08 04:35:31 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:31 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:31.367167 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:31.360790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.66:48955)
0808 04:35:31.361203 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:31.361218 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:31.361374 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:31.361642 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:31.361645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:31.361890 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:31.362000 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:31.362005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:31.362791 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:31.362796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:31.363440 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:31.363448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:31.363926 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:31.363930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:31.364014 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:31.364555 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:31.364567 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:31.367011 (+ 2444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":115,"threads_started":1}
W20260808 04:35:31.367223 29256 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:31.360966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:48955 (local address 127.15.228.126:32887)
0808 04:35:31.361169 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:31.361174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:31.361308 (+ 134us) server_negotiation.cc:407] Connection header received
0808 04:35:31.361482 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:31.361485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:31.361535 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:31.361621 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:31.362125 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:31.362677 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:31.363567 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:31.363792 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:31.363911 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:31.367126 (+ 3215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.66:48955: BlockingRecv error: recv got EOF from 127.15.228.66:48955 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 08 04:35:31 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:31 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:31.603839 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:31.596621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.67:54933)
0808 04:35:31.597225 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:31.597252 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:31.597439 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:31.597765 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:31.597768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:31.598018 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:31.598124 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:31.598130 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:31.599047 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:31.599053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:31.599696 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:31.599704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:31.600237 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:31.600241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:31.600350 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:31.600927 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:31.600942 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:31.603620 (+ 2678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":132,"threads_started":1}
W20260808 04:35:31.603852 29256 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:31.596835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:54933 (local address 127.15.228.126:32887)
0808 04:35:31.597173 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:31.597179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:31.597361 (+ 182us) server_negotiation.cc:407] Connection header received
0808 04:35:31.597557 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:31.597563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:31.597629 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:31.597736 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:31.598265 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:31.598869 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:31.599844 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:31.600093 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:31.600213 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:31.603750 (+ 3537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.67:54933: BlockingRecv error: recv got EOF from 127.15.228.67:54933 (error 108)
Metrics: {"server-negotiator.queue_time_us":206}
Aug 08 04:35:32 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:32 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:32.178385 29261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:32.171422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.65:56635)
0808 04:35:32.171910 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:32.171927 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:32.172049 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:32.172638 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:32.172641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:32.172843 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:32.172960 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:32.172966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:32.173947 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:32.173949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:32.174529 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:32.174538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:32.175486 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:32.175489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:32.175565 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:32.176204 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:32.176219 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:32.178239 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":155,"threads_started":1}
W20260808 04:35:32.178563 29262 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:32.171693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:56635 (local address 127.15.228.126:32887)
0808 04:35:32.172003 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:32.172007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:32.172028 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:35:32.172317 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:32.172321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:32.172422 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:32.172530 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:32.173193 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:32.173770 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:32.174820 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:32.175301 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:32.175417 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:32.178416 (+ 2999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.65:56635: BlockingRecv error: recv got EOF from 127.15.228.65:56635 (error 108)
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":87,"threads_started":1}
Aug 08 04:35:32 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:32 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:32.374292 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:32.368094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.66:38727)
0808 04:35:32.368382 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:32.368396 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:32.368482 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:32.368878 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:32.368880 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:32.369073 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:32.369184 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:32.369189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:32.370073 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:32.370076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:32.370633 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:32.370641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:32.371420 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:32.371423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:32.371487 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:32.372082 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:32.372094 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:32.374169 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":101,"threads_started":1}
W20260808 04:35:32.374768 29262 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:32.368225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:38727 (local address 127.15.228.126:32887)
0808 04:35:32.368489 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:32.368494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:32.368566 (+ 72us) server_negotiation.cc:407] Connection header received
0808 04:35:32.368622 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:32.368624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:32.368714 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:32.368807 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:32.369367 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:32.369934 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:32.370822 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:32.371275 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:32.371382 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:32.374571 (+ 3189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.66:38727: BlockingRecv error: recv got EOF from 127.15.228.66:38727 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 08 04:35:32 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:32 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:32.611264 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:32.604965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.67:34495)
0808 04:35:32.605289 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:32.605303 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:32.605408 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:32.605720 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:32.605723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:32.605915 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:32.606025 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:32.606031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:32.606967 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:32.606970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:32.607545 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:32.607553 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:32.608056 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:32.608060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:32.608147 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:32.608728 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:32.608740 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:32.611061 (+ 2321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":136,"threads_started":1}
W20260808 04:35:32.611307 29262 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:32.605157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:34495 (local address 127.15.228.126:32887)
0808 04:35:32.605318 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:32.605322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:32.605359 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:35:32.605500 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:32.605502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:32.605555 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:32.605635 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:32.606175 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:32.606756 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:32.607677 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:32.607900 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:32.608052 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:32.611188 (+ 3136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.67:34495: BlockingRecv error: recv got EOF from 127.15.228.67:34495 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 08 04:35:33 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:33 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:33.187129 29268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:33.179769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.65:41567)
0808 04:35:33.180267 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:33.180284 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:33.180392 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:33.180808 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:33.180811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:33.181069 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:33.181357 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:33.181367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:33.182466 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:33.182469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:33.183107 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:33.183116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:33.183798 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:33.183802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:33.183872 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:33.184557 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:33.184571 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:33.186954 (+ 2383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":259,"threads_started":1}
W20260808 04:35:33.187285 29271 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:33.180009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:41567 (local address 127.15.228.126:32887)
0808 04:35:33.180345 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:33.180349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:33.180366 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:35:33.180550 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:33.180552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:33.180606 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:33.180694 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:33.181570 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:33.182312 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:33.183322 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:33.183604 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:33.183740 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:33.187145 (+ 3405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.65:41567: BlockingRecv error: recv got EOF from 127.15.228.65:41567 (error 108)
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":82,"threads_started":1}
Aug 08 04:35:33 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:33 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:33.382372 29272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:33.375506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.66:40045)
0808 04:35:33.375956 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:33.375971 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:33.376163 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:33.376476 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:33.376478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:33.376722 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:33.376850 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:33.376856 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:33.377806 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:33.377809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:33.378487 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:33.378496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:33.379170 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:33.379174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:33.379252 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:33.380015 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:33.380035 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:33.382216 (+ 2181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":162,"threads_started":1}
W20260808 04:35:33.382756 29271 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:33.375706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:40045 (local address 127.15.228.126:32887)
0808 04:35:33.375912 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:33.375917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:33.376110 (+ 193us) server_negotiation.cc:407] Connection header received
0808 04:35:33.376270 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:33.376273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:33.376342 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:33.376423 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:33.376998 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:33.377661 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:33.378638 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:33.378877 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:33.379013 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:33.382611 (+ 3598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.66:40045: BlockingRecv error: recv got EOF from 127.15.228.66:40045 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 08 04:35:33 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:33 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:33.619455 29273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:33.612271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.67:42185)
0808 04:35:33.612710 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:33.612725 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:33.612888 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:33.613197 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:33.613200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:33.613437 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:33.613538 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:33.613545 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:33.614502 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:33.614509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:33.615273 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:33.615295 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:33.615993 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:33.615999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:33.616177 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:33.617038 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:33.617054 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:33.619325 (+ 2271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":116,"threads_started":1}
W20260808 04:35:33.619710 29271 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:33.612451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:42185 (local address 127.15.228.126:32887)
0808 04:35:33.612627 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:33.612633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:33.612817 (+ 184us) server_negotiation.cc:407] Connection header received
0808 04:35:33.613011 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:33.613014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:33.613080 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:33.613176 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:33.613689 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:33.614372 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:33.615423 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:33.615714 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:33.615897 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:33.619588 (+ 3691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.67:42185: BlockingRecv error: recv got EOF from 127.15.228.67:42185 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 08 04:35:34 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:34 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:34.200016 29274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:34.188026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.65:33101)
0808 04:35:34.188298 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:34.188320 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:34.188414 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:34.188665 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:34.188667 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:34.188856 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:34.188986 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:34.188992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:34.189811 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:34.189814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:34.190510 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:34.190522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:34.191579 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:34.191590 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:34.191712 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:34.196367 (+ 4655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:34.196387 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:34.199869 (+ 3482us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":90,"threads_started":1}
W20260808 04:35:34.200343 29275 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:34.188123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:33101 (local address 127.15.228.126:32887)
0808 04:35:34.188387 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:34.188390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:34.188403 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:35:34.188503 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:34.188505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:34.188551 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:34.188629 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:34.189118 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:34.189667 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:34.190692 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:34.191323 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:34.191443 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:34.200140 (+ 8697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.65:33101: BlockingRecv error: recv got EOF from 127.15.228.65:33101 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
Aug 08 04:35:34 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:34 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:34.390025 29276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:34.383448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.66:57037)
0808 04:35:34.383781 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:34.383796 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:34.383915 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:34.384237 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:34.384240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:34.384455 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:34.384566 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:34.384572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:34.385579 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:34.385584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:34.386201 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:34.386226 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:34.386702 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:34.386706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:34.386814 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:34.387753 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:34.387772 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:34.389896 (+ 2124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":134,"threads_started":1}
W20260808 04:35:34.390329 29275 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:34.383531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:57037 (local address 127.15.228.126:32887)
0808 04:35:34.383790 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:34.383794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:34.383898 (+ 104us) server_negotiation.cc:407] Connection header received
0808 04:35:34.384001 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:34.384003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:34.384052 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:34.384131 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:34.384696 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:34.385228 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:34.386343 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:34.386578 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:34.386688 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:34.390132 (+ 3444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.66:57037: BlockingRecv error: recv got EOF from 127.15.228.66:57037 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
Aug 08 04:35:34 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:34 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:34.627650 29277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:34.620388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.67:34151)
0808 04:35:34.620830 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:34.620849 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:34.621035 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:34.621389 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:34.621392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:34.621649 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:34.621779 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:34.621785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:34.622717 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:34.622721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:34.623349 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:34.623358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:34.624092 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:34.624095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:34.624167 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:34.624952 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:34.624967 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:34.627469 (+ 2502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":85,"threads_started":1}
W20260808 04:35:34.627739 29275 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:34.620581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:34151 (local address 127.15.228.126:32887)
0808 04:35:34.620782 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:34.620786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:34.621040 (+ 254us) server_negotiation.cc:407] Connection header received
0808 04:35:34.621121 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:34.621123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:34.621189 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:34.621283 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:34.621944 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:34.622551 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:34.623548 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:34.623899 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:34.624015 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:34.627615 (+ 3600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.67:34151: BlockingRecv error: recv got EOF from 127.15.228.67:34151 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 08 04:35:35 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:35 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:35.208163 29280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:35.201340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.65:54797)
0808 04:35:35.201683 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:35.201697 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:35.201835 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:35.202329 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:35.202332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:35.202608 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:35.202755 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:35.202763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:35.203818 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:35.203821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:35.204499 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:35.204510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:35.205048 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:35.205051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:35.205118 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:35.205692 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:35.205704 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:35.208029 (+ 2325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":71,"threads_started":1}
W20260808 04:35:35.208457 29279 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:35.201229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:54797 (local address 127.15.228.126:32887)
0808 04:35:35.201561 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:35.201565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:35.201776 (+ 211us) server_negotiation.cc:407] Connection header received
0808 04:35:35.202010 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:35.202015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:35.202084 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:35.202309 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:35.202974 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:35.203661 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:35.204646 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:35.204903 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:35.204996 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:35.208191 (+ 3195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.65:54797: BlockingRecv error: recv got EOF from 127.15.228.65:54797 (error 108)
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":104,"threads_started":1}
Aug 08 04:35:35 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:35 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:35.397311 29282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:35.390904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.66:59011)
0808 04:35:35.391224 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:35.391237 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:35.391347 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:35.391616 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:35.391619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:35.391819 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:35.391919 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:35.391924 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:35.392737 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:35.392740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:35.393303 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:35.393310 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:35.393894 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:35.393899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:35.394011 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:35.394806 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:35.394819 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:35.397173 (+ 2354us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":110,"threads_started":1}
W20260808 04:35:35.397527 29279 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:35.391108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:59011 (local address 127.15.228.126:32887)
0808 04:35:35.391258 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:35.391262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:35.391301 (+ 39us) server_negotiation.cc:407] Connection header received
0808 04:35:35.391429 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:35.391432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:35.391477 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:35.391551 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:35.392066 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:35.392607 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:35.393431 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:35.393657 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:35.393744 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:35.397324 (+ 3580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.66:59011: BlockingRecv error: recv got EOF from 127.15.228.66:59011 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 04:35:35 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:35 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:35.634446 29283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:35.628805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.67:53027)
0808 04:35:35.629121 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:35.629136 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:35.629252 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:35.629521 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:35.629524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:35.629712 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:35.629815 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:35.629821 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:35.630618 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:35.630620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:35.631309 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:35.631319 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:35.631810 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:35.631814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:35.631880 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:35.632456 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:35.632468 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:35.634306 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":132,"threads_started":1}
W20260808 04:35:35.634616 29279 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:35.628921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:53027 (local address 127.15.228.126:32887)
0808 04:35:35.629074 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:35.629078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:35.629247 (+ 169us) server_negotiation.cc:407] Connection header received
0808 04:35:35.629328 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:35.629330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:35.629384 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:35.629460 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:35.629939 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:35.630498 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:35.631446 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:35.631689 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:35.631781 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:35.634473 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.67:53027: BlockingRecv error: recv got EOF from 127.15.228.67:53027 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 08 04:35:36 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:36 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:36.215972 29284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:36.209148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.65:53597)
0808 04:35:36.209498 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:36.209514 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:36.209629 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:36.209966 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:36.209969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:36.210174 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:36.210281 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:36.210287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:36.211410 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:36.211417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:36.212077 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:36.212113 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:36.212922 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:36.212927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:36.213011 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:36.213884 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:36.213896 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:36.215852 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":101,"threads_started":1}
W20260808 04:35:36.216212 29285 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:36.209208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:53597 (local address 127.15.228.126:32887)
0808 04:35:36.209567 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:36.209570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:36.209585 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:35:36.209772 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:36.209774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:36.209825 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:36.209936 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:36.210570 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:36.211265 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:36.212334 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:36.212754 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:36.212900 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:36.216064 (+ 3164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.65:53597: BlockingRecv error: recv got EOF from 127.15.228.65:53597 (error 108)
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":103,"threads_started":1}
Aug 08 04:35:36 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:36 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:36.405081 29286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:36.398263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.66:48155)
0808 04:35:36.398707 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:36.398721 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:36.398845 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:36.399192 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:36.399195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:36.399400 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:36.399500 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:36.399505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:36.400585 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:36.400588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:36.401208 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:36.401216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:36.401770 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:36.401774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:36.401847 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:36.402508 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:36.402522 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:36.404882 (+ 2360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
W20260808 04:35:36.405182 29285 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:36.398557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:48155 (local address 127.15.228.126:32887)
0808 04:35:36.398741 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:36.398745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:36.398803 (+ 58us) server_negotiation.cc:407] Connection header received
0808 04:35:36.398996 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:36.398998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:36.399048 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:36.399128 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:36.399745 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:36.400398 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:36.401370 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:36.401628 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:36.401727 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:36.405057 (+ 3330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.66:48155: BlockingRecv error: recv got EOF from 127.15.228.66:48155 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 08 04:35:36 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:36 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.67@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:36.642647 29287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:36.635764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.67:35787)
0808 04:35:36.636171 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:36.636189 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:36.636351 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:36.636697 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:36.636701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:36.636940 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:36.637075 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:36.637082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:36.638014 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:36.638018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:36.639012 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:36.639029 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:36.639724 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:36.639731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:36.639840 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:36.640566 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:36.640578 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:36.642495 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":104,"threads_started":1}
W20260808 04:35:36.643016 29285 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:36.635979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:35787 (local address 127.15.228.126:32887)
0808 04:35:36.636178 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:36.636182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:36.636325 (+ 143us) server_negotiation.cc:407] Connection header received
0808 04:35:36.636453 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:36.636455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:36.636514 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:36.636601 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:36.637242 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:36.637872 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:36.639178 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:36.639525 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:36.639689 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:36.642821 (+ 3132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.67:35787: BlockingRecv error: recv got EOF from 127.15.228.67:35787 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 08 04:35:37 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:37 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.65@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:37.224308 29289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:37.216954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.65:40627)
0808 04:35:37.217275 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:37.217291 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:37.217392 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:37.217657 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:37.217660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:37.217855 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:37.217972 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:37.217977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:37.219292 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:37.219296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:37.220189 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:37.220201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:37.220991 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:37.220994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:37.221062 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:37.221761 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:37.221772 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:37.224160 (+ 2388us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":93,"threads_started":1}
W20260808 04:35:37.224532 29291 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:37.217076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:40627 (local address 127.15.228.126:32887)
0808 04:35:37.217314 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:37.217317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:37.217346 (+ 29us) server_negotiation.cc:407] Connection header received
0808 04:35:37.217486 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:37.217488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:37.217535 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:37.217615 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:37.218107 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:37.218910 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:37.220362 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:37.220774 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:37.220891 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:37.224349 (+ 3458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.65:40627: BlockingRecv error: recv got EOF from 127.15.228.65:40627 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":63,"threads_started":1}
Aug 08 04:35:37 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 04:35:37 dist-test-slave-drl0 krb5kdc[27841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.228.66@KRBTEST.COM for oryx/127.15.228.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 04:35:37.412804 29293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:37.406214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:32887 (local address 127.15.228.66:48195)
0808 04:35:37.406534 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:37.406549 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:37.406675 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:37.406986 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:37.406989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:37.407261 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:37.407370 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:37.407376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:37.408204 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:37.408207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:37.408804 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:37.408813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:37.409383 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:37.409387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:37.409517 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:37.410135 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:37.410148 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:37.412594 (+ 2446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.228.126:32887: Server oryx/127.15.228.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":122,"threads_started":1}
W20260808 04:35:37.412849 29291 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:37.406374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:48195 (local address 127.15.228.126:32887)
0808 04:35:37.406525 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:37.406529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:37.406647 (+ 118us) server_negotiation.cc:407] Connection header received
0808 04:35:37.406808 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:37.406811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:37.406859 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:37.406966 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:37.407523 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:37.408083 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:37.408946 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:37.409169 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:37.409326 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:37.412731 (+ 3405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.228.66:48195: BlockingRecv error: recv got EOF from 127.15.228.66:48195 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:35:37.510762 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 28637
I20260808 04:35:37.528782 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 28844
I20260808 04:35:37.544159 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 29024
I20260808 04:35:37.559748 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 28540
2026-08-08T04:35:37Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18614 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T04:35:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:35:37Z Disabled control of system clock
I20260808 04:35:37.600281 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:34597
--webserver_interface=127.15.228.126
--webserver_port=0
--builtin_ntp_servers=127.15.228.84:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:34597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 04:35:37.735498 29299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:37.735802 29299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:37.735841 29299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:37.741577 29299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:35:37.741657 29299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:37.741674 29299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:37.741693 29299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:35:37.741708 29299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:35:37.747475 29299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:34597
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:34597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:37.748730 29299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:37.750202 29299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:37.762308 29305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:37.762327 29304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:37.762377 29299 server_base.cc:1034] running on GCE node
W20260808 04:35:37.762317 29308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:37.764343 29299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:37.765458 29299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:37.766651 29299 hybrid_clock.cc:648] HybridClock initialized: now 1786163737766619 us; error 47 us; skew 500 ppm
I20260808 04:35:37.768654 29299 webserver.cc:460] Webserver started at http://127.15.228.126:39511/ using document root <none> and password file <none>
I20260808 04:35:37.769213 29299 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:37.769277 29299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:37.769518 29299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:37.771881 29299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "2cd72b9c78e9493a9d6969e843d86978"
format_stamp: "Formatted at 2026-08-08 04:35:37 on dist-test-slave-drl0"
server_key: "45bc142ce62b4168bdb0a20d33873e9d"
server_key_iv: "66ca9887d0aca9e0cab0776188c4c772"
server_key_version: "encryptionkey@0"
I20260808 04:35:37.772435 29299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "2cd72b9c78e9493a9d6969e843d86978"
format_stamp: "Formatted at 2026-08-08 04:35:37 on dist-test-slave-drl0"
server_key: "45bc142ce62b4168bdb0a20d33873e9d"
server_key_iv: "66ca9887d0aca9e0cab0776188c4c772"
server_key_version: "encryptionkey@0"
I20260808 04:35:37.777045 29299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:35:37.780506 29316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:37.782021 29299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:35:37.782137 29299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "2cd72b9c78e9493a9d6969e843d86978"
format_stamp: "Formatted at 2026-08-08 04:35:37 on dist-test-slave-drl0"
server_key: "45bc142ce62b4168bdb0a20d33873e9d"
server_key_iv: "66ca9887d0aca9e0cab0776188c4c772"
server_key_version: "encryptionkey@0"
I20260808 04:35:37.782239 29299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:37.796275 29299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:37.797420 29299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:37.797619 29299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:37.805480 29299 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:34597
I20260808 04:35:37.805516 29384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:34597 every 8 connection(s)
I20260808 04:35:37.806458 29299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:35:37.810061 29385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:37.811183 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 29299
I20260808 04:35:37.811285 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:35:37.811619 16273 external_mini_cluster.cc:1428] Setting key 6f963e06cc016b42979a882719ad14b7
I20260808 04:35:37.817241 29385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978: Bootstrap starting.
I20260808 04:35:37.820242 29385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:37.821410 29385 log.cc:824] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:37.823125 29390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:37.813099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36878 (local address 127.15.228.126:34597)
0808 04:35:37.813711 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:37.813727 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:37.813762 (+ 35us) server_negotiation.cc:407] Connection header received
0808 04:35:37.814410 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:37.814426 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:37.815001 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:37.815311 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:37.815796 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:37.816758 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:37.818184 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:37.819197 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:37.819423 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:37.820561 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:37.820567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:37.820579 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:35:37.821116 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:37.821181 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:35:37.821205 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:37.821300 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:37.821847 (+ 547us) server_negotiation.cc:299] Negotiation successful
0808 04:35:37.822305 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":142,"threads_started":1}
I20260808 04:35:37.824005 29385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978: No bootstrap required, opened a new log
I20260808 04:35:37.827921 29385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd72b9c78e9493a9d6969e843d86978" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34597 } }
I20260808 04:35:37.828140 29385 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:37.828182 29385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd72b9c78e9493a9d6969e843d86978, State: Initialized, Role: FOLLOWER
I20260808 04:35:37.828588 29385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd72b9c78e9493a9d6969e843d86978" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34597 } }
I20260808 04:35:37.828708 29385 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:35:37.828748 29385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:35:37.828812 29385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:37.829692 29385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd72b9c78e9493a9d6969e843d86978" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34597 } }
I20260808 04:35:37.829990 29385 leader_election.cc:302] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cd72b9c78e9493a9d6969e843d86978; no voters:
I20260808 04:35:37.830277 29385 leader_election.cc:288] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:35:37.830461 29392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:35:37.830711 29392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978 [term 1 LEADER]: Becoming Leader. State: Replica: 2cd72b9c78e9493a9d6969e843d86978, State: Running, Role: LEADER
I20260808 04:35:37.831089 29392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd72b9c78e9493a9d6969e843d86978" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34597 } }
I20260808 04:35:37.831549 29385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:35:37.832788 29393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cd72b9c78e9493a9d6969e843d86978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd72b9c78e9493a9d6969e843d86978" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34597 } } }
I20260808 04:35:37.832775 29394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cd72b9c78e9493a9d6969e843d86978. Latest consensus state: current_term: 1 leader_uuid: "2cd72b9c78e9493a9d6969e843d86978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd72b9c78e9493a9d6969e843d86978" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34597 } } }
I20260808 04:35:37.832921 29393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:37.832927 29394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:37.833513 29407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:35:37.836675 29407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:35:37.842634 29407 catalog_manager.cc:1347] Generated new cluster ID: 79363262d6e446f29244fec9690955d2
I20260808 04:35:37.842762 29407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:35:37.866814 29407 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:35:37.868085 29407 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:35:37.877461 29407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cd72b9c78e9493a9d6969e843d86978: Generated new TSK 0
I20260808 04:35:37.878291 29407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:35:37.887173 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:34597
--builtin_ntp_servers=127.15.228.84:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 04:35:38.020659 29419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:38.020925 29419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:38.020982 29419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:38.026518 29419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:38.026589 29419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:38.026648 29419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:35:38.033193 29419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:34597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:38.034364 29419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:38.035853 29419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:38.046103 29425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:38.046070 29429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:38.046247 29426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:38.046921 29419 server_base.cc:1034] running on GCE node
I20260808 04:35:38.047464 29419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:38.048156 29419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:38.049373 29419 hybrid_clock.cc:648] HybridClock initialized: now 1786163738049334 us; error 55 us; skew 500 ppm
I20260808 04:35:38.051532 29419 webserver.cc:460] Webserver started at http://127.15.228.65:44131/ using document root <none> and password file <none>
I20260808 04:35:38.052142 29419 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:38.052240 29419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:38.052480 29419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:38.054596 29419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "85a45c31c30347e1b88705eb4e2abda5"
format_stamp: "Formatted at 2026-08-08 04:35:38 on dist-test-slave-drl0"
server_key: "154d1939432f1e5c967a7da6ebcf750b"
server_key_iv: "a46936ac25aab769fee1b24cbaa8a282"
server_key_version: "encryptionkey@0"
I20260808 04:35:38.055197 29419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "85a45c31c30347e1b88705eb4e2abda5"
format_stamp: "Formatted at 2026-08-08 04:35:38 on dist-test-slave-drl0"
server_key: "154d1939432f1e5c967a7da6ebcf750b"
server_key_iv: "a46936ac25aab769fee1b24cbaa8a282"
server_key_version: "encryptionkey@0"
I20260808 04:35:38.058847 29419 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:35:38.062116 29439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:38.063746 29419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 04:35:38.063923 29419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "85a45c31c30347e1b88705eb4e2abda5"
format_stamp: "Formatted at 2026-08-08 04:35:38 on dist-test-slave-drl0"
server_key: "154d1939432f1e5c967a7da6ebcf750b"
server_key_iv: "a46936ac25aab769fee1b24cbaa8a282"
server_key_version: "encryptionkey@0"
I20260808 04:35:38.064050 29419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:38.090241 29419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:38.091223 29419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:38.091451 29419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:38.092123 29419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:38.093158 29419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:38.093238 29419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:38.093312 29419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:38.093365 29419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:38.103415 29419 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:38149
I20260808 04:35:38.103443 29599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:38149 every 8 connection(s)
I20260808 04:35:38.104377 29419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:35:38.107038 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 29419
I20260808 04:35:38.107173 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:35:38.107518 16273 external_mini_cluster.cc:1428] Setting key 3f67331369053476bc50578cc1e55f21
I20260808 04:35:38.109611 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:34597
--builtin_ntp_servers=127.15.228.84:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 04:35:38.116650 29390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:38.106371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:60237 (local address 127.15.228.126:34597)
0808 04:35:38.106554 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:38.106558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:38.107853 (+ 1295us) server_negotiation.cc:407] Connection header received
0808 04:35:38.109469 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:38.109475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:38.109539 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:38.109645 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:38.111344 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:38.112140 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:38.113692 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:38.114124 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:38.114279 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:38.115491 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:38.115497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:38.115501 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:35:38.115521 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:38.115548 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:35:38.115562 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:38.115660 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:38.116014 (+ 354us) server_negotiation.cc:299] Negotiation successful
0808 04:35:38.116333 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:35:38.117054 29605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:38.106778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34597 (local address 127.15.228.65:60237)
0808 04:35:38.107698 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:38.107741 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:38.109261 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:38.110154 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:38.110166 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:38.110299 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:38.111179 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:38.111200 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:38.112362 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:38.112368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:38.113529 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:38.113540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:38.114315 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:38.114319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:38.114487 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:38.115209 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:35:38.115210 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:38.115250 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:35:38.115322 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:38.115694 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:38.115704 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:38.115709 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:35:38.115822 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 04:35:38.116146 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":151,"threads_started":1}
I20260808 04:35:38.118491 29601 heartbeater.cc:347] Connected to a master server at 127.15.228.126:34597
I20260808 04:35:38.118745 29601 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:38.119360 29601 heartbeater.cc:511] Master 127.15.228.126:34597 requested a full tablet report, sending...
I20260808 04:35:38.120915 29337 ts_manager.cc:194] Registered new tserver with Master: 85a45c31c30347e1b88705eb4e2abda5 (127.15.228.65:38149)
I20260808 04:35:38.122118 29337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.228.65:60237
W20260808 04:35:38.238744 29606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:38.239039 29606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:38.239080 29606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:38.243283 29606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:38.243438 29606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:38.243502 29606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:35:38.248488 29606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:34597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:38.249604 29606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:38.251070 29606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:38.261436 29619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:38.261546 29617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:38.261628 29606 server_base.cc:1034] running on GCE node
W20260808 04:35:38.261476 29616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:38.262161 29606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:38.262830 29606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:38.264061 29606 hybrid_clock.cc:648] HybridClock initialized: now 1786163738264000 us; error 84 us; skew 500 ppm
I20260808 04:35:38.266155 29606 webserver.cc:460] Webserver started at http://127.15.228.66:39871/ using document root <none> and password file <none>
I20260808 04:35:38.266759 29606 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:38.266856 29606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:38.267122 29606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:38.269263 29606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "2cb7a014ef40499b9e14a3479eade70f"
format_stamp: "Formatted at 2026-08-08 04:35:38 on dist-test-slave-drl0"
server_key: "b668f67e70f6a81bcbe4ff4b224681ee"
server_key_iv: "4c20d59ef03b4f18a00064566c209f75"
server_key_version: "encryptionkey@0"
I20260808 04:35:38.269843 29606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "2cb7a014ef40499b9e14a3479eade70f"
format_stamp: "Formatted at 2026-08-08 04:35:38 on dist-test-slave-drl0"
server_key: "b668f67e70f6a81bcbe4ff4b224681ee"
server_key_iv: "4c20d59ef03b4f18a00064566c209f75"
server_key_version: "encryptionkey@0"
I20260808 04:35:38.273835 29606 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:35:38.276618 29626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:38.278472 29606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:35:38.278640 29606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "2cb7a014ef40499b9e14a3479eade70f"
format_stamp: "Formatted at 2026-08-08 04:35:38 on dist-test-slave-drl0"
server_key: "b668f67e70f6a81bcbe4ff4b224681ee"
server_key_iv: "4c20d59ef03b4f18a00064566c209f75"
server_key_version: "encryptionkey@0"
I20260808 04:35:38.278770 29606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:38.292061 29606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:38.293167 29606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:38.293468 29606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:38.294160 29606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:38.295415 29606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:38.295487 29606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:38.295554 29606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:38.295605 29606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:38.306315 29606 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:33925
I20260808 04:35:38.306337 29787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:33925 every 8 connection(s)
I20260808 04:35:38.307332 29606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:35:38.308465 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 29606
I20260808 04:35:38.308590 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:35:38.308938 16273 external_mini_cluster.cc:1428] Setting key 9c42dc545adc8231e1ced561086cabc4
I20260808 04:35:38.311324 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:34597
--builtin_ntp_servers=127.15.228.84:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 04:35:38.320328 29390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:38.309624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:56685 (local address 127.15.228.126:34597)
0808 04:35:38.309849 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:38.309861 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:38.311267 (+ 1406us) server_negotiation.cc:407] Connection header received
0808 04:35:38.313609 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:38.313615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:38.313683 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:38.313795 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:38.315486 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:38.316012 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:38.317989 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:38.318207 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:38.318325 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:38.319808 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:38.319811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:38.319813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:35:38.319826 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:38.319846 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:35:38.319855 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:38.319926 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:38.320150 (+ 224us) server_negotiation.cc:299] Negotiation successful
0808 04:35:38.320210 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 04:35:38.321692 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:38.310147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34597 (local address 127.15.228.66:56685)
0808 04:35:38.311051 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:38.311116 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:38.313295 (+ 2179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:38.313985 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:38.314093 (+ 108us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:38.314249 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:38.315291 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:38.315319 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:38.316162 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:38.316168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:38.317833 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:38.317850 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:38.318395 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:38.318402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:38.318677 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:38.319447 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:35:38.319451 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:38.319515 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:35:38.319615 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:38.319956 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:38.319968 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:38.319977 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 04:35:38.320123 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 04:35:38.320479 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":164,"threads_started":1}
I20260808 04:35:38.323436 29791 heartbeater.cc:347] Connected to a master server at 127.15.228.126:34597
I20260808 04:35:38.323701 29791 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:38.324271 29791 heartbeater.cc:511] Master 127.15.228.126:34597 requested a full tablet report, sending...
I20260808 04:35:38.325412 29337 ts_manager.cc:194] Registered new tserver with Master: 2cb7a014ef40499b9e14a3479eade70f (127.15.228.66:33925)
I20260808 04:35:38.326023 29337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.228.66:56685
W20260808 04:35:38.437134 29794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:38.437425 29794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:38.437463 29794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:38.441658 29794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:38.441720 29794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:38.441773 29794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:35:38.447008 29794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:34597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:38.448343 29794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:38.449918 29794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:38.461377 29805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:38.461488 29803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:38.461715 29794 server_base.cc:1034] running on GCE node
W20260808 04:35:38.461377 29801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:38.462239 29794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:38.462904 29794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:38.464253 29794 hybrid_clock.cc:648] HybridClock initialized: now 1786163738464217 us; error 26 us; skew 500 ppm
I20260808 04:35:38.466904 29794 webserver.cc:460] Webserver started at http://127.15.228.67:46725/ using document root <none> and password file <none>
I20260808 04:35:38.467621 29794 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:38.467691 29794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:38.467931 29794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:38.470577 29794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "4d7c20bad4da403aa9e859a7a24841b8"
format_stamp: "Formatted at 2026-08-08 04:35:38 on dist-test-slave-drl0"
server_key: "6812d9755d260abee7e36b2f0f480202"
server_key_iv: "98861e48cced8081a9dc6ba520b70d4c"
server_key_version: "encryptionkey@0"
I20260808 04:35:38.471184 29794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "4d7c20bad4da403aa9e859a7a24841b8"
format_stamp: "Formatted at 2026-08-08 04:35:38 on dist-test-slave-drl0"
server_key: "6812d9755d260abee7e36b2f0f480202"
server_key_iv: "98861e48cced8081a9dc6ba520b70d4c"
server_key_version: "encryptionkey@0"
I20260808 04:35:38.475723 29794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260808 04:35:38.479147 29814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:38.480618 29794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:35:38.480738 29794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "4d7c20bad4da403aa9e859a7a24841b8"
format_stamp: "Formatted at 2026-08-08 04:35:38 on dist-test-slave-drl0"
server_key: "6812d9755d260abee7e36b2f0f480202"
server_key_iv: "98861e48cced8081a9dc6ba520b70d4c"
server_key_version: "encryptionkey@0"
I20260808 04:35:38.480914 29794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:38.495203 29794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:38.496443 29794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:38.496754 29794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:38.497648 29794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:38.499118 29794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:38.499188 29794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:38.499240 29794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:38.499301 29794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:38.510473 29794 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:37203
I20260808 04:35:38.510494 29971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:37203 every 8 connection(s)
I20260808 04:35:38.511452 29794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:35:38.520336 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 29794
I20260808 04:35:38.520440 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:35:38.520733 16273 external_mini_cluster.cc:1428] Setting key 4238f35f770c2094cdc9410525622828
I20260808 04:35:38.524187 29390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:38.514310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:35577 (local address 127.15.228.126:34597)
0808 04:35:38.514550 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:38.514555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:38.516152 (+ 1597us) server_negotiation.cc:407] Connection header received
0808 04:35:38.518132 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:38.518136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:38.518207 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:38.518292 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:38.519444 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:38.520015 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:38.521565 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:38.521825 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:38.521946 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:38.523437 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:38.523440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:38.523442 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:35:38.523462 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:38.523482 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:35:38.523489 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:38.523564 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:38.524001 (+ 437us) server_negotiation.cc:299] Negotiation successful
0808 04:35:38.524062 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260808 04:35:38.525545 29975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:38.514875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34597 (local address 127.15.228.67:35577)
0808 04:35:38.515844 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:38.515904 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:38.517844 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:38.518482 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:38.518490 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:38.518583 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:38.519227 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:38.519262 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:38.520153 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:38.520156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:38.521432 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:38.521443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:38.522032 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:38.522037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:38.522248 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:38.523084 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:35:38.523088 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:38.523149 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:35:38.523249 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:38.523676 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:38.523695 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:38.523705 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 04:35:38.523909 (+ 204us) client_negotiation.cc:240] Negotiation successful
0808 04:35:38.524406 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":145,"threads_started":1}
I20260808 04:35:38.527349 29973 heartbeater.cc:347] Connected to a master server at 127.15.228.126:34597
I20260808 04:35:38.527743 29973 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:38.528520 29973 heartbeater.cc:511] Master 127.15.228.126:34597 requested a full tablet report, sending...
I20260808 04:35:38.530237 29337 ts_manager.cc:194] Registered new tserver with Master: 4d7c20bad4da403aa9e859a7a24841b8 (127.15.228.67:37203)
I20260808 04:35:38.531185 29337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.228.67:35577
I20260808 04:35:38.535990 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 04:35:38.539405 29390 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:38.538318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36884 (local address 127.15.228.126:34597)
0808 04:35:38.538521 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:38.538526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:38.538667 (+ 141us) server_negotiation.cc:407] Connection header received
0808 04:35:38.538834 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:38.538837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:38.538888 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:38.539006 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:38.539255 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36884: BlockingRecv error: recv got EOF from 127.0.0.1:36884 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
I20260808 04:35:38.539964 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 29419
I20260808 04:35:38.567950 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 29606
I20260808 04:35:38.582556 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 29794
I20260808 04:35:38.594143 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 29299
2026-08-08T04:35:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1036 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T04:35:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:35:38Z Disabled control of system clock
I20260808 04:35:38.636229 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:37761
--webserver_interface=127.15.228.126
--webserver_port=0
--builtin_ntp_servers=127.15.228.84:35909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:37761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 04:35:38.759613 29988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:38.760027 29988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:38.760090 29988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:38.764132 29988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:35:38.764189 29988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:38.764204 29988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:38.764222 29988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:35:38.764237 29988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:35:38.768973 29988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:35909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:37761
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:37761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:38.770051 29988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:38.771519 29988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:38.781486 29994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:38.781621 29995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:38.781635 29988 server_base.cc:1034] running on GCE node
W20260808 04:35:38.781869 29997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:38.782810 29988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:38.784190 29988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:38.785429 29988 hybrid_clock.cc:648] HybridClock initialized: now 1786163738785387 us; error 61 us; skew 500 ppm
I20260808 04:35:38.788219 29988 webserver.cc:460] Webserver started at http://127.15.228.126:37251/ using document root <none> and password file <none>
I20260808 04:35:38.789225 29988 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:38.789389 29988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:38.789683 29988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:38.792074 29988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "46e692ea87f64625a8fad693d7b442f9"
format_stamp: "Formatted at 2026-08-08 04:35:38 on dist-test-slave-drl0"
server_key: "6f15f959e0817808e0855dd6d688e558"
server_key_iv: "d8ab51d07d6ce57bcec158b47ae995a6"
server_key_version: "encryptionkey@0"
I20260808 04:35:38.792707 29988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "46e692ea87f64625a8fad693d7b442f9"
format_stamp: "Formatted at 2026-08-08 04:35:38 on dist-test-slave-drl0"
server_key: "6f15f959e0817808e0855dd6d688e558"
server_key_iv: "d8ab51d07d6ce57bcec158b47ae995a6"
server_key_version: "encryptionkey@0"
I20260808 04:35:38.796911 29988 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:35:38.800238 30005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:38.801862 29988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:35:38.802038 29988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "46e692ea87f64625a8fad693d7b442f9"
format_stamp: "Formatted at 2026-08-08 04:35:38 on dist-test-slave-drl0"
server_key: "6f15f959e0817808e0855dd6d688e558"
server_key_iv: "d8ab51d07d6ce57bcec158b47ae995a6"
server_key_version: "encryptionkey@0"
I20260808 04:35:38.802166 29988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:38.807137 29988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:38.807789 29988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:38.807988 29988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:38.815668 29988 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:37761
I20260808 04:35:38.815689 30079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:37761 every 8 connection(s)
I20260808 04:35:38.816624 29988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:35:38.819944 30080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:38.824863 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 29988
I20260808 04:35:38.824990 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:35:38.825292 16273 external_mini_cluster.cc:1428] Setting key 453fd373caab5222caaf77fcfca2cf72
I20260808 04:35:38.826823 30080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9: Bootstrap starting.
I20260808 04:35:38.829288 30080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:38.831218 30080 log.cc:824] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:38.831665 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:38.826887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49708 (local address 127.15.228.126:37761)
0808 04:35:38.827582 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:38.827597 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:38.827632 (+ 35us) server_negotiation.cc:407] Connection header received
0808 04:35:38.828260 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:38.828276 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:38.828622 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:38.828955 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:38.830128 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:38.830133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:38.830145 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:35:38.830156 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:38.830212 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:35:38.830228 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:38.830357 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:38.830509 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 04:35:38.830757 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":130,"threads_started":1}
I20260808 04:35:38.833544 30080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9: No bootstrap required, opened a new log
I20260808 04:35:38.837651 30080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e692ea87f64625a8fad693d7b442f9" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 37761 } }
I20260808 04:35:38.837873 30080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:38.837915 30080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e692ea87f64625a8fad693d7b442f9, State: Initialized, Role: FOLLOWER
I20260808 04:35:38.838328 30080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e692ea87f64625a8fad693d7b442f9" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 37761 } }
I20260808 04:35:38.838454 30080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:35:38.838495 30080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:35:38.838560 30080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:38.839514 30080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e692ea87f64625a8fad693d7b442f9" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 37761 } }
I20260808 04:35:38.839807 30080 leader_election.cc:302] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46e692ea87f64625a8fad693d7b442f9; no voters:
I20260808 04:35:38.840108 30080 leader_election.cc:288] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:35:38.840520 30086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:35:38.840886 30086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9 [term 1 LEADER]: Becoming Leader. State: Replica: 46e692ea87f64625a8fad693d7b442f9, State: Running, Role: LEADER
I20260808 04:35:38.841383 30080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:35:38.841378 30086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e692ea87f64625a8fad693d7b442f9" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 37761 } }
I20260808 04:35:38.843765 30088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46e692ea87f64625a8fad693d7b442f9. Latest consensus state: current_term: 1 leader_uuid: "46e692ea87f64625a8fad693d7b442f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e692ea87f64625a8fad693d7b442f9" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 37761 } } }
I20260808 04:35:38.844445 30088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9 [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:38.844131 30087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46e692ea87f64625a8fad693d7b442f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e692ea87f64625a8fad693d7b442f9" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 37761 } } }
I20260808 04:35:38.844775 30087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9 [sys.catalog]: This master's current role is: LEADER
W20260808 04:35:38.846902 30104 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:35:38.847046 30104 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:35:38.847121 30105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:35:38.847833 30105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:35:38.853696 30105 catalog_manager.cc:1347] Generated new cluster ID: c852d6d56e7c401fb1419c96a76a7612
I20260808 04:35:38.853789 30105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:35:38.874584 30105 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:35:38.874989 30105 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:35:38.888351 30105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46e692ea87f64625a8fad693d7b442f9: Generated new TSK 0
I20260808 04:35:38.889334 30105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:35:38.904842 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:37761
--builtin_ntp_servers=127.15.228.84:35909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 04:35:39.049899 30109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:39.050184 30109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:39.050226 30109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:39.054325 30109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:39.054389 30109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:39.054451 30109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:35:39.059346 30109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:35909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:37761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:39.060667 30109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:39.062268 30109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:39.074105 30122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:39.074280 30124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:39.074360 30120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:39.074318 30109 server_base.cc:1034] running on GCE node
I20260808 04:35:39.075289 30109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:39.075997 30109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:39.077373 30109 hybrid_clock.cc:648] HybridClock initialized: now 1786163739077305 us; error 73 us; skew 500 ppm
I20260808 04:35:39.079711 30109 webserver.cc:460] Webserver started at http://127.15.228.65:40811/ using document root <none> and password file <none>
I20260808 04:35:39.080350 30109 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:39.080448 30109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:39.080711 30109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:39.083232 30109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "ce36f4cb6ed2472ea424bdef1194da7c"
format_stamp: "Formatted at 2026-08-08 04:35:39 on dist-test-slave-drl0"
server_key: "ef51ef2ed9eede798a0a4394c83d17e1"
server_key_iv: "815407aed8f44bd2f7ab9e5a64bf7cc5"
server_key_version: "encryptionkey@0"
I20260808 04:35:39.083923 30109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "ce36f4cb6ed2472ea424bdef1194da7c"
format_stamp: "Formatted at 2026-08-08 04:35:39 on dist-test-slave-drl0"
server_key: "ef51ef2ed9eede798a0a4394c83d17e1"
server_key_iv: "815407aed8f44bd2f7ab9e5a64bf7cc5"
server_key_version: "encryptionkey@0"
I20260808 04:35:39.088807 30109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 04:35:39.093138 30132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:39.094749 30109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:35:39.094988 30109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "ce36f4cb6ed2472ea424bdef1194da7c"
format_stamp: "Formatted at 2026-08-08 04:35:39 on dist-test-slave-drl0"
server_key: "ef51ef2ed9eede798a0a4394c83d17e1"
server_key_iv: "815407aed8f44bd2f7ab9e5a64bf7cc5"
server_key_version: "encryptionkey@0"
I20260808 04:35:39.095180 30109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:39.100175 30109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:39.100838 30109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:39.101052 30109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:39.101687 30109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:39.102681 30109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:39.102761 30109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:39.102838 30109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:39.102881 30109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:39.113046 30109 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:36635
I20260808 04:35:39.113059 30286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:36635 every 8 connection(s)
I20260808 04:35:39.114147 30109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:35:39.116143 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 30109
I20260808 04:35:39.116278 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:35:39.116607 16273 external_mini_cluster.cc:1428] Setting key c57bc504f3c4f453a02069bee2173dcb
I20260808 04:35:39.118993 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:37761
--builtin_ntp_servers=127.15.228.84:35909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 04:35:39.122330 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:39.116255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:45391 (local address 127.15.228.126:37761)
0808 04:35:39.116420 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:39.116426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:39.117666 (+ 1240us) server_negotiation.cc:407] Connection header received
0808 04:35:39.119441 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:39.119444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:39.119482 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:39.119559 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:39.121490 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:39.121498 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:39.121505 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:35:39.121535 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:39.121584 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:35:39.121623 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:39.121779 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:39.122012 (+ 233us) server_negotiation.cc:299] Negotiation successful
0808 04:35:39.122092 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:35:39.123286 30291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:39.116725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:37761 (local address 127.15.228.65:45391)
0808 04:35:39.117542 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:39.117582 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:39.119189 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:39.119739 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:39.119750 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:39.119867 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:39.120735 (+ 868us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:35:39.120738 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:39.120802 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:35:39.121149 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:39.121813 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:39.121824 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:39.121830 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 04:35:39.122012 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 04:35:39.122218 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":168,"threads_started":1}
I20260808 04:35:39.124928 30288 heartbeater.cc:347] Connected to a master server at 127.15.228.126:37761
I20260808 04:35:39.125386 30288 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:39.126222 30288 heartbeater.cc:511] Master 127.15.228.126:37761 requested a full tablet report, sending...
I20260808 04:35:39.129011 30029 ts_manager.cc:194] Registered new tserver with Master: ce36f4cb6ed2472ea424bdef1194da7c (127.15.228.65:36635)
W20260808 04:35:39.251447 30293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:39.251739 30293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:39.251786 30293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:39.256031 30293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:39.256093 30293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:39.256151 30293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:35:39.261451 30293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:35909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:37761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:39.262615 30293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:39.265033 30293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:39.277138 30300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:39.277161 30303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:39.277387 30293 server_base.cc:1034] running on GCE node
W20260808 04:35:39.277169 30301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:39.277981 30293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:39.278687 30293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:39.279877 30293 hybrid_clock.cc:648] HybridClock initialized: now 1786163739279863 us; error 38 us; skew 500 ppm
I20260808 04:35:39.288612 30293 webserver.cc:460] Webserver started at http://127.15.228.66:42751/ using document root <none> and password file <none>
I20260808 04:35:39.289286 30293 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:39.289361 30293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:39.289568 30293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:39.292091 30293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "dcccbc5005684d5e910da29c115f8d1e"
format_stamp: "Formatted at 2026-08-08 04:35:39 on dist-test-slave-drl0"
server_key: "bab931958e5267a903463c5a07ab9b38"
server_key_iv: "7013992242f097d3367f041d11c0330e"
server_key_version: "encryptionkey@0"
I20260808 04:35:39.292681 30293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "dcccbc5005684d5e910da29c115f8d1e"
format_stamp: "Formatted at 2026-08-08 04:35:39 on dist-test-slave-drl0"
server_key: "bab931958e5267a903463c5a07ab9b38"
server_key_iv: "7013992242f097d3367f041d11c0330e"
server_key_version: "encryptionkey@0"
I20260808 04:35:39.296836 30293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:35:39.300400 30309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:39.302134 30293 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:35:39.302518 30293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "dcccbc5005684d5e910da29c115f8d1e"
format_stamp: "Formatted at 2026-08-08 04:35:39 on dist-test-slave-drl0"
server_key: "bab931958e5267a903463c5a07ab9b38"
server_key_iv: "7013992242f097d3367f041d11c0330e"
server_key_version: "encryptionkey@0"
I20260808 04:35:39.303004 30293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:39.309005 30293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:39.309666 30293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:39.309867 30293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:39.310526 30293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:39.311571 30293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:39.311647 30293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:39.311722 30293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:39.311772 30293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:39.322194 30293 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:37705
I20260808 04:35:39.322348 30469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:37705 every 8 connection(s)
I20260808 04:35:39.323282 30293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:35:39.330650 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:39.325575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:55219 (local address 127.15.228.126:37761)
0808 04:35:39.325751 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:39.325755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:39.326840 (+ 1085us) server_negotiation.cc:407] Connection header received
0808 04:35:39.328260 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:39.328263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:39.328305 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:39.328399 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:39.329747 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:39.329750 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:39.329753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:35:39.329764 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:39.329784 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:35:39.329792 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:39.329882 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:39.330351 (+ 469us) server_negotiation.cc:299] Negotiation successful
0808 04:35:39.330430 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 04:35:39.331003 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 30293
I20260808 04:35:39.331158 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:35:39.331524 16273 external_mini_cluster.cc:1428] Setting key 90931bbfa4784d83296c16702d81b112
I20260808 04:35:39.332145 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:39.325969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:37761 (local address 127.15.228.66:55219)
0808 04:35:39.326679 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:39.326721 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:39.328014 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:39.328636 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:39.328643 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:39.328732 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:39.329453 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:35:39.329455 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:39.329497 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:35:39.329568 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:39.330002 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:39.330023 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:39.330034 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 04:35:39.330328 (+ 294us) client_negotiation.cc:240] Negotiation successful
0808 04:35:39.330981 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":119,"threads_started":1}
I20260808 04:35:39.333799 30471 heartbeater.cc:347] Connected to a master server at 127.15.228.126:37761
I20260808 04:35:39.334154 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:37761
--builtin_ntp_servers=127.15.228.84:35909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 04:35:39.334254 30471 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:39.334885 30471 heartbeater.cc:511] Master 127.15.228.126:37761 requested a full tablet report, sending...
I20260808 04:35:39.336495 30029 ts_manager.cc:194] Registered new tserver with Master: dcccbc5005684d5e910da29c115f8d1e (127.15.228.66:37705)
W20260808 04:35:39.471323 30474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:39.471653 30474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:39.471758 30474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:39.477587 30474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:39.477670 30474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:39.477872 30474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:35:39.485308 30474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:35909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:37761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:39.486562 30474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:39.488308 30474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:39.498467 30481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:39.498467 30480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:39.498517 30474 server_base.cc:1034] running on GCE node
W20260808 04:35:39.498457 30483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:39.499182 30474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:39.499874 30474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:39.501194 30474 hybrid_clock.cc:648] HybridClock initialized: now 1786163739501151 us; error 42 us; skew 500 ppm
I20260808 04:35:39.503844 30474 webserver.cc:460] Webserver started at http://127.15.228.67:39955/ using document root <none> and password file <none>
I20260808 04:35:39.504508 30474 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:39.504606 30474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:39.504844 30474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:39.507165 30474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "cee1123a5a6c48dead84c17f7a22d328"
format_stamp: "Formatted at 2026-08-08 04:35:39 on dist-test-slave-drl0"
server_key: "8ca3b630d1ebdc52c0a1f6c4e3222387"
server_key_iv: "a185ba448bcdc123ee054e92d96de3ae"
server_key_version: "encryptionkey@0"
I20260808 04:35:39.507871 30474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "cee1123a5a6c48dead84c17f7a22d328"
format_stamp: "Formatted at 2026-08-08 04:35:39 on dist-test-slave-drl0"
server_key: "8ca3b630d1ebdc52c0a1f6c4e3222387"
server_key_iv: "a185ba448bcdc123ee054e92d96de3ae"
server_key_version: "encryptionkey@0"
I20260808 04:35:39.512210 30474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:35:39.516038 30491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:39.517318 30474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260808 04:35:39.517470 30474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "cee1123a5a6c48dead84c17f7a22d328"
format_stamp: "Formatted at 2026-08-08 04:35:39 on dist-test-slave-drl0"
server_key: "8ca3b630d1ebdc52c0a1f6c4e3222387"
server_key_iv: "a185ba448bcdc123ee054e92d96de3ae"
server_key_version: "encryptionkey@0"
I20260808 04:35:39.517606 30474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:39.522475 30474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:39.523226 30474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:39.523422 30474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:39.524046 30474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:39.524991 30474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:39.525063 30474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:39.525131 30474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:39.525182 30474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:39.535928 30474 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:36817
I20260808 04:35:39.535953 30649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:36817 every 8 connection(s)
I20260808 04:35:39.536990 30474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:35:39.544070 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:39.539276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:56059 (local address 127.15.228.126:37761)
0808 04:35:39.539540 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:39.539545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:39.540542 (+ 997us) server_negotiation.cc:407] Connection header received
0808 04:35:39.542005 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:39.542009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:39.542064 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:39.542169 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:39.543552 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:39.543555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:39.543557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:35:39.543568 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:39.543587 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:35:39.543596 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:39.543701 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:39.543864 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 04:35:39.543918 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260808 04:35:39.544502 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 30474
I20260808 04:35:39.544641 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:35:39.544950 16273 external_mini_cluster.cc:1428] Setting key a6899c1afbc1f678ea8bdceec90809ad
I20260808 04:35:39.545217 30652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:39.539600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:37761 (local address 127.15.228.67:56059)
0808 04:35:39.540359 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:39.540402 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:39.541775 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:39.542377 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:39.542387 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:39.542489 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:39.543228 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:35:39.543230 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:39.543271 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:35:39.543341 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:39.543696 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:39.543706 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:39.543713 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 04:35:39.543912 (+ 199us) client_negotiation.cc:240] Negotiation successful
0808 04:35:39.544151 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":137,"threads_started":1}
I20260808 04:35:39.547098 30650 heartbeater.cc:347] Connected to a master server at 127.15.228.126:37761
I20260808 04:35:39.547544 30650 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:39.548302 30650 heartbeater.cc:511] Master 127.15.228.126:37761 requested a full tablet report, sending...
I20260808 04:35:39.549862 30028 ts_manager.cc:194] Registered new tserver with Master: cee1123a5a6c48dead84c17f7a22d328 (127.15.228.67:36817)
I20260808 04:35:39.561352 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 04:35:39.565498 30083 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:35:39.564180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49714 (local address 127.15.228.126:37761)
0808 04:35:39.564334 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:39.564339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:39.564482 (+ 143us) server_negotiation.cc:407] Connection header received
0808 04:35:39.564840 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:39.564843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:39.564891 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:39.564990 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:39.565379 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49714: BlockingRecv error: recv got EOF from 127.0.0.1:49714 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:35:39.565697 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 30109
I20260808 04:35:39.577380 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 30293
I20260808 04:35:39.596421 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 30474
I20260808 04:35:39.610121 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 29988
2026-08-08T04:35:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (1010 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:35:39 dist-test-slave-drl0 krb5kdc[30662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:35:39 dist-test-slave-drl0 krb5kdc[30662](info): set up 2 sockets
Aug 08 04:35:39 dist-test-slave-drl0 krb5kdc[30662](info): commencing operation
krb5kdc: starting...
W20260808 04:35:40.942904 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.307s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:35:41 dist-test-slave-drl0 krb5kdc[30662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:35:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:35:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:35:41.108306 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:40429
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:42089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:40429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:41.266465 30681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:41.266866 30681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:41.267025 30681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:41.271854 30681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:35:41.272044 30681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:41.272087 30681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:41.272119 30681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:35:41.272158 30681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:35:41.279642 30681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:42089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:40429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:40429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:41.281348 30681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:41.283499 30681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:41.293726 30686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:41.293862 30687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:41.294116 30689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:41.294456 30681 server_base.cc:1034] running on GCE node
I20260808 04:35:41.295701 30681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:41.298653 30681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:41.299921 30681 hybrid_clock.cc:648] HybridClock initialized: now 1786163741299868 us; error 89 us; skew 500 ppm
Aug 08 04:35:41 dist-test-slave-drl0 krb5kdc[30662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163741, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:41.303759 30681 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:35:41.304862 30681 webserver.cc:460] Webserver started at http://127.15.228.126:37749/ using document root <none> and password file <none>
I20260808 04:35:41.305488 30681 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:41.305584 30681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:41.305828 30681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:41.309259 30681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "4f270d8d34b1442bba3c423c3e206dca"
format_stamp: "Formatted at 2026-08-08 04:35:41 on dist-test-slave-drl0"
server_key: "6838428be32d89f1d045974f8245a405"
server_key_iv: "e2ae7eb6dc93adf2dd1ab1c7262eebd7"
server_key_version: "encryptionkey@0"
I20260808 04:35:41.310312 30681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "4f270d8d34b1442bba3c423c3e206dca"
format_stamp: "Formatted at 2026-08-08 04:35:41 on dist-test-slave-drl0"
server_key: "6838428be32d89f1d045974f8245a405"
server_key_iv: "e2ae7eb6dc93adf2dd1ab1c7262eebd7"
server_key_version: "encryptionkey@0"
I20260808 04:35:41.316720 30681 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260808 04:35:41.321276 30699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:41.323338 30681 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:35:41.323513 30681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "4f270d8d34b1442bba3c423c3e206dca"
format_stamp: "Formatted at 2026-08-08 04:35:41 on dist-test-slave-drl0"
server_key: "6838428be32d89f1d045974f8245a405"
server_key_iv: "e2ae7eb6dc93adf2dd1ab1c7262eebd7"
server_key_version: "encryptionkey@0"
I20260808 04:35:41.323657 30681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:41.363240 30681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:41.367411 30681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:41.367645 30681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:41.379413 30681 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:40429
I20260808 04:35:41.380720 30781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:40429 every 8 connection(s)
I20260808 04:35:41.381981 30681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:35:41.385854 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:41.389743 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 30681
I20260808 04:35:41.389909 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:35:41.390267 16273 external_mini_cluster.cc:1428] Setting key 421268a1c907a3dbfa6fbd65a86f8e2f
I20260808 04:35:41.393687 30782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca: Bootstrap starting.
I20260808 04:35:41.396304 30782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:41.397164 30782 log.cc:824] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:41.400319 30782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca: No bootstrap required, opened a new log
Aug 08 04:35:41 dist-test-slave-drl0 krb5kdc[30662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:35:41.405959 30782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f270d8d34b1442bba3c423c3e206dca" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 40429 } }
I20260808 04:35:41.406213 30782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:41.406327 30782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f270d8d34b1442bba3c423c3e206dca, State: Initialized, Role: FOLLOWER
I20260808 04:35:41.406812 30782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f270d8d34b1442bba3c423c3e206dca" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 40429 } }
I20260808 04:35:41.406997 30782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:35:41.407096 30782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:35:41.407230 30782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:41.408453 30782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f270d8d34b1442bba3c423c3e206dca" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 40429 } }
I20260808 04:35:41.408923 30782 leader_election.cc:302] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f270d8d34b1442bba3c423c3e206dca; no voters:
I20260808 04:35:41.409328 30782 leader_election.cc:288] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:35:41.409606 30793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:35:41.409888 30793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca [term 1 LEADER]: Becoming Leader. State: Replica: 4f270d8d34b1442bba3c423c3e206dca, State: Running, Role: LEADER
I20260808 04:35:41.410240 30793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f270d8d34b1442bba3c423c3e206dca" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 40429 } }
I20260808 04:35:41.411007 30782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:35:41.412098 30791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:41.392865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.15.228.126:40429)
0808 04:35:41.393512 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:41.393529 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:41.393572 (+ 43us) server_negotiation.cc:407] Connection header received
0808 04:35:41.394463 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:41.394485 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:41.394857 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:41.395323 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:41.396345 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:41.397665 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:41.399335 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:41.400289 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:41.400520 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:41.405186 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:41.405193 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:41.405210 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:41.405250 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:41.407668 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:41.408212 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:41.408225 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:41.408231 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:41.408361 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:41.408706 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:41.408710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:41.408712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:41.409377 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:41.410369 (+ 992us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:41.410543 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 04:35:41.410902 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":184,"threads_started":1}
I20260808 04:35:41.413242 30795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f270d8d34b1442bba3c423c3e206dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f270d8d34b1442bba3c423c3e206dca" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 40429 } } }
I20260808 04:35:41.413427 30795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:41.413242 30796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f270d8d34b1442bba3c423c3e206dca. Latest consensus state: current_term: 1 leader_uuid: "4f270d8d34b1442bba3c423c3e206dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f270d8d34b1442bba3c423c3e206dca" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 40429 } } }
I20260808 04:35:41.413793 30796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca [sys.catalog]: This master's current role is: LEADER
I20260808 04:35:41.414137 30805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:35:41.419030 30805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:35:41.427209 30805 catalog_manager.cc:1347] Generated new cluster ID: 1b8ebd18b74342ada91c7c8a4c238bc0
I20260808 04:35:41.427313 30805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:35:41.445305 30805 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:35:41.446372 30805 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:35:41.455646 30805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca: Generated new TSK 0
I20260808 04:35:41.456851 30805 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
I20260808 04:35:41.503829 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:40429
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:42089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:41.678041 30818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:41.678614 30818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:41.678732 30818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:41.686331 30818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:41.686446 30818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:41.686548 30818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:35:41.693696 30818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:42089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:40429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:41.696192 30818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:41.698769 30818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:35:41.712222 30818 server_base.cc:1034] running on GCE node
W20260808 04:35:41.712245 30823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:41.712280 30824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:41.712390 30826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:41.713980 30818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:41.715289 30818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:41.716747 30818 hybrid_clock.cc:648] HybridClock initialized: now 1786163741716692 us; error 68 us; skew 500 ppm
Aug 08 04:35:41 dist-test-slave-drl0 krb5kdc[30662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163741, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:41.721098 30818 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:35:41.722220 30818 webserver.cc:460] Webserver started at http://127.15.228.65:40277/ using document root <none> and password file <none>
I20260808 04:35:41.723122 30818 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:41.723241 30818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:41.723512 30818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:41.726078 30818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "8c89ae6b601b4cb0ab5c06db86c3bae8"
format_stamp: "Formatted at 2026-08-08 04:35:41 on dist-test-slave-drl0"
server_key: "6293a5a25022011693ac55ed94e88a22"
server_key_iv: "0cc04cb14ec31abc3aba4f9b302eb700"
server_key_version: "encryptionkey@0"
I20260808 04:35:41.726742 30818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "8c89ae6b601b4cb0ab5c06db86c3bae8"
format_stamp: "Formatted at 2026-08-08 04:35:41 on dist-test-slave-drl0"
server_key: "6293a5a25022011693ac55ed94e88a22"
server_key_iv: "0cc04cb14ec31abc3aba4f9b302eb700"
server_key_version: "encryptionkey@0"
I20260808 04:35:41.731835 30818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 04:35:41.735968 30836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:41.738029 30818 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:35:41.738397 30818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "8c89ae6b601b4cb0ab5c06db86c3bae8"
format_stamp: "Formatted at 2026-08-08 04:35:41 on dist-test-slave-drl0"
server_key: "6293a5a25022011693ac55ed94e88a22"
server_key_iv: "0cc04cb14ec31abc3aba4f9b302eb700"
server_key_version: "encryptionkey@0"
I20260808 04:35:41.738584 30818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:41.761598 30818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:41.766137 30818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:41.766402 30818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:41.767160 30818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:41.768107 30818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:41.768184 30818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:41.768270 30818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:41.768326 30818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:41.779446 30818 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:46257
I20260808 04:35:41.779536 30991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:46257 every 8 connection(s)
I20260808 04:35:41.781018 30818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:35:41.786163 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 30818
I20260808 04:35:41.786314 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:35:41.786710 16273 external_mini_cluster.cc:1428] Setting key 48b98f887a082b3cb9867fc7bec2a008
Aug 08 04:35:41 dist-test-slave-drl0 krb5kdc[30662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163741, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.228.66@KRBTEST.COM; defaulting to no policy
I20260808 04:35:41.802918 30791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:41.783557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:41125 (local address 127.15.228.126:40429)
0808 04:35:41.783741 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:41.783746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:41.784656 (+ 910us) server_negotiation.cc:407] Connection header received
0808 04:35:41.786537 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:41.786540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:41.786595 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:41.786678 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:41.788288 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:41.789082 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:41.791268 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:41.791808 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:41.792009 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:41.797089 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:41.797094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:41.797096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:41.797130 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:41.799143 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:41.800379 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:41.800384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:41.800385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:41.800501 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:41.801354 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:41.801358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:41.801360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:41.801690 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:41.801834 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:41.802552 (+ 718us) server_negotiation.cc:299] Negotiation successful
0808 04:35:41.802714 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 04:35:41.803975 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:41.783902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:40429 (local address 127.15.228.65:41125)
0808 04:35:41.784476 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:41.784526 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:41.786192 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:41.786917 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:41.786970 (+ 53us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:41.787497 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:41.788124 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:41.788140 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:41.789265 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:41.789271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:41.790989 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:41.791007 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:41.792149 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:41.792154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:41.792388 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:41.793308 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:41.793328 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:41.796749 (+ 3421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:41.799336 (+ 2587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:41.799348 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:41.799378 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:41.800160 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:41.800781 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:41.800785 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:41.800788 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:41.801136 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:41.801866 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:41.801876 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:41.802243 (+ 367us) client_negotiation.cc:769] Sending connection context
0808 04:35:41.802488 (+ 245us) client_negotiation.cc:240] Negotiation successful
0808 04:35:41.802911 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":152,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.66@KRBTEST.COM" created.
I20260808 04:35:41.805338 30992 heartbeater.cc:347] Connected to a master server at 127.15.228.126:40429
I20260808 04:35:41.805626 30992 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:41.806169 30992 heartbeater.cc:511] Master 127.15.228.126:40429 requested a full tablet report, sending...
I20260808 04:35:41.807941 30727 ts_manager.cc:194] Registered new tserver with Master: 8c89ae6b601b4cb0ab5c06db86c3bae8 (127.15.228.65:46257)
I20260808 04:35:41.809618 30727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:41125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
I20260808 04:35:41.828024 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:40429
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:42089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:41.976131 30998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:41.976456 30998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:41.976547 30998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:41.981093 30998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:41.981186 30998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:41.981279 30998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:35:41.986778 30998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:42089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:40429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:41.988651 30998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:41.989809 30998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:42.003329 31010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:42.003362 31005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:42.003383 31006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:42.003645 30998 server_base.cc:1034] running on GCE node
I20260808 04:35:42.004213 30998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:42.005179 30998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:42.006418 30998 hybrid_clock.cc:648] HybridClock initialized: now 1786163742006381 us; error 52 us; skew 500 ppm
Aug 08 04:35:42 dist-test-slave-drl0 krb5kdc[30662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163742, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:42.010138 30998 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:35:42.011456 30998 webserver.cc:460] Webserver started at http://127.15.228.66:45917/ using document root <none> and password file <none>
I20260808 04:35:42.012302 30998 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:42.012418 30998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:42.012682 30998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:42.015684 30998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "306764445d4f42189d6c4299f3bbfb57"
format_stamp: "Formatted at 2026-08-08 04:35:42 on dist-test-slave-drl0"
server_key: "c32137beac002ee786df4d3b7f1c8012"
server_key_iv: "459e7bdcfa48b41bc8fa6fac0f4f4634"
server_key_version: "encryptionkey@0"
I20260808 04:35:42.016325 30998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "306764445d4f42189d6c4299f3bbfb57"
format_stamp: "Formatted at 2026-08-08 04:35:42 on dist-test-slave-drl0"
server_key: "c32137beac002ee786df4d3b7f1c8012"
server_key_iv: "459e7bdcfa48b41bc8fa6fac0f4f4634"
server_key_version: "encryptionkey@0"
I20260808 04:35:42.022195 30998 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260808 04:35:42.025333 31019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:42.027148 30998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:35:42.027302 30998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "306764445d4f42189d6c4299f3bbfb57"
format_stamp: "Formatted at 2026-08-08 04:35:42 on dist-test-slave-drl0"
server_key: "c32137beac002ee786df4d3b7f1c8012"
server_key_iv: "459e7bdcfa48b41bc8fa6fac0f4f4634"
server_key_version: "encryptionkey@0"
I20260808 04:35:42.027436 30998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:42.040742 30998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:42.045372 30998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:42.045627 30998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:42.046556 30998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:42.047564 30998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:42.047640 30998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:42.047716 30998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:42.047770 30998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:42.061342 30998 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:42255
I20260808 04:35:42.061440 31191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:42255 every 8 connection(s)
I20260808 04:35:42.062510 30998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:35:42.069123 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 30998
I20260808 04:35:42.069283 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:35:42.069592 16273 external_mini_cluster.cc:1428] Setting key e90b1d94862a04cdacf567115536aa38
Aug 08 04:35:42 dist-test-slave-drl0 krb5kdc[30662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163742, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:35:42.082643 30791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:42.064851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:43541 (local address 127.15.228.126:40429)
0808 04:35:42.065039 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:42.065043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:42.066011 (+ 968us) server_negotiation.cc:407] Connection header received
0808 04:35:42.067656 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:42.067660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:42.067727 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:42.067851 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:42.069200 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.069941 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.071761 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.072037 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.072176 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:42.076832 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:42.076836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:42.076839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:42.076868 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:42.079154 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:42.080063 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:42.080067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:42.080068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:42.080219 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:42.080940 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:42.080958 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:42.080960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:42.081324 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:42.081438 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:42.082345 (+ 907us) server_negotiation.cc:299] Negotiation successful
0808 04:35:42.082502 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 04:35:42.083951 31194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:42.065208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:40429 (local address 127.15.228.66:43541)
0808 04:35:42.065738 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:42.065785 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:42.067410 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:42.068048 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:42.068056 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:42.068486 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:42.069030 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:42.069045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.070161 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.070164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:42.071481 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:42.071492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.072230 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.072234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:42.072420 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:42.073296 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:42.073313 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:42.076592 (+ 3279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:42.079394 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:42.079406 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:42.079431 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:42.079867 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:42.080366 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:42.080372 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:42.080376 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:42.080718 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:42.081488 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:42.081499 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:42.081970 (+ 471us) client_negotiation.cc:769] Sending connection context
0808 04:35:42.082236 (+ 266us) client_negotiation.cc:240] Negotiation successful
0808 04:35:42.082714 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":148,"threads_started":1}
I20260808 04:35:42.085520 31192 heartbeater.cc:347] Connected to a master server at 127.15.228.126:40429
I20260808 04:35:42.085807 31192 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:42.086376 31192 heartbeater.cc:511] Master 127.15.228.126:40429 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.228.67@KRBTEST.COM; defaulting to no policy
I20260808 04:35:42.087584 30727 ts_manager.cc:194] Registered new tserver with Master: 306764445d4f42189d6c4299f3bbfb57 (127.15.228.66:42255)
I20260808 04:35:42.088217 30727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:43541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
I20260808 04:35:42.112475 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:40429
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:42089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:35:42.262015 31197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:35:42.262387 31197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:35:42.262450 31197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:35:42.268242 31197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:35:42.268323 31197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:35:42.268389 31197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:35:42.274680 31197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:42089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:40429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:35:42.275873 31197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:35:42.276918 31197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:35:42.289572 31204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:42.289616 31203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:35:42.290272 31208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:35:42.290407 31197 server_base.cc:1034] running on GCE node
I20260808 04:35:42.291709 31197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:35:42.292820 31197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:35:42.294057 31197 hybrid_clock.cc:648] HybridClock initialized: now 1786163742294040 us; error 43 us; skew 500 ppm
Aug 08 04:35:42 dist-test-slave-drl0 krb5kdc[30662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163742, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:35:42.297631 31197 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:35:42.298753 31197 webserver.cc:460] Webserver started at http://127.15.228.67:37753/ using document root <none> and password file <none>
I20260808 04:35:42.299396 31197 fs_manager.cc:359] Metadata directory not provided
I20260808 04:35:42.299465 31197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:35:42.299750 31197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:35:42.301987 31197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "104ccac8503e465cb2941d8765464ebe"
format_stamp: "Formatted at 2026-08-08 04:35:42 on dist-test-slave-drl0"
server_key: "51dd4577565123058f27b28bc0fdf881"
server_key_iv: "f1b0174bd6947dd980ad956cc10ed9b5"
server_key_version: "encryptionkey@0"
I20260808 04:35:42.302593 31197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "104ccac8503e465cb2941d8765464ebe"
format_stamp: "Formatted at 2026-08-08 04:35:42 on dist-test-slave-drl0"
server_key: "51dd4577565123058f27b28bc0fdf881"
server_key_iv: "f1b0174bd6947dd980ad956cc10ed9b5"
server_key_version: "encryptionkey@0"
I20260808 04:35:42.306741 31197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260808 04:35:42.310333 31220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:42.312402 31197 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:35:42.312561 31197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "104ccac8503e465cb2941d8765464ebe"
format_stamp: "Formatted at 2026-08-08 04:35:42 on dist-test-slave-drl0"
server_key: "51dd4577565123058f27b28bc0fdf881"
server_key_iv: "f1b0174bd6947dd980ad956cc10ed9b5"
server_key_version: "encryptionkey@0"
I20260808 04:35:42.312729 31197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:35:42.339923 31197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:35:42.345281 31197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:35:42.345594 31197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:35:42.346495 31197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:35:42.347719 31197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:35:42.347806 31197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:42.347892 31197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:35:42.347939 31197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:35:42.358700 31197 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:36541
I20260808 04:35:42.358829 31384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:36541 every 8 connection(s)
I20260808 04:35:42.359782 31197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:35:42.364461 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 31197
I20260808 04:35:42.364574 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:35:42.364883 16273 external_mini_cluster.cc:1428] Setting key 7bf76f5d7c7b092fa50d98a1ead7d2ab
Aug 08 04:35:42 dist-test-slave-drl0 krb5kdc[30662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163742, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:35:42.376333 30791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:42.362059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:43717 (local address 127.15.228.126:40429)
0808 04:35:42.362255 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:42.362259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:42.363444 (+ 1185us) server_negotiation.cc:407] Connection header received
0808 04:35:42.365151 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:42.365154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:42.365216 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:42.365332 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:42.366840 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.367477 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.368846 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.369131 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.369263 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:42.372217 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:42.372221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:42.372223 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:42.372251 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:42.374035 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:42.374594 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:42.374598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:42.374599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:42.374692 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:42.375121 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:42.375126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:42.375128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:42.375350 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:42.375466 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:42.376008 (+ 542us) server_negotiation.cc:299] Negotiation successful
0808 04:35:42.376146 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 04:35:42.377244 31388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:42.362447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:40429 (local address 127.15.228.67:43717)
0808 04:35:42.363271 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:42.363318 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:42.364892 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:42.365509 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:42.365516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:42.366119 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:35:42.366646 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:42.366660 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.367634 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.367640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:42.368564 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:42.368574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.369361 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.369364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:42.369521 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:42.370205 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:35:42.370222 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:35:42.372007 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:35:42.374182 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:42.374191 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:42.374210 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:42.374479 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:42.374799 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:35:42.374804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:35:42.374811 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:35:42.375016 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:35:42.375465 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:35:42.375472 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:35:42.375750 (+ 278us) client_negotiation.cc:769] Sending connection context
0808 04:35:42.375952 (+ 202us) client_negotiation.cc:240] Negotiation successful
0808 04:35:42.376296 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":125,"threads_started":1}
I20260808 04:35:42.378657 31385 heartbeater.cc:347] Connected to a master server at 127.15.228.126:40429
I20260808 04:35:42.379011 31385 heartbeater.cc:464] Registering TS with master...
I20260808 04:35:42.379621 31385 heartbeater.cc:511] Master 127.15.228.126:40429 requested a full tablet report, sending...
I20260808 04:35:42.380888 30725 ts_manager.cc:194] Registered new tserver with Master: 104ccac8503e465cb2941d8765464ebe (127.15.228.67:36541)
I20260808 04:35:42.381695 30725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:43717
I20260808 04:35:42.392174 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:35:42.404219 30791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:42.395085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33636 (local address 127.15.228.126:40429)
0808 04:35:42.395367 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:42.395372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:42.395501 (+ 129us) server_negotiation.cc:407] Connection header received
0808 04:35:42.395578 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:42.395580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:42.395630 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:42.395706 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:35:42.396705 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.397320 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.398440 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.398759 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.398856 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:42.400713 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:35:42.400716 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:35:42.400718 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:35:42.400742 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:35:42.402239 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:42.402719 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:42.402721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:42.402722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:42.402805 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:35:42.403277 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:35:42.403280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:35:42.403285 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:35:42.403545 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:35:42.403659 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:42.403990 (+ 331us) server_negotiation.cc:299] Negotiation successful
0808 04:35:42.404097 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260808 04:35:42.408524 30725 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:35:42.412592 30725 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:35:42.431725 31403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:42.425091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:42255 (local address 127.0.0.1:60068)
0808 04:35:42.425830 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:42.425844 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:42.425941 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:42.426518 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:42.426522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:42.426543 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:42.426691 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:42.426698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.428315 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.428318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:42.430172 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:42.430186 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.431283 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.431287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:42.431382 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:42.431405 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:35:42.431473 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 04:35:42.431553 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":304,"thread_start_us":118,"threads_started":1}
I20260808 04:35:42.431784 31404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:42.425610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60068 (local address 127.15.228.66:42255)
0808 04:35:42.426046 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:42.426060 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:42.426086 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:35:42.426164 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:42.426168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:42.426343 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:42.426525 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:42.426861 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.428135 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.430339 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.431170 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.431279 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:42.431471 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:42.431526 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 04:35:42.431623 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":141,"threads_started":1}
I20260808 04:35:42.431736 31399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:42.424306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:36541 (local address 127.0.0.1:58870)
0808 04:35:42.425219 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:42.425279 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:42.425432 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:42.426287 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:42.426294 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:42.426326 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:42.426532 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:42.426541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.428326 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.428329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:42.429900 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:42.429910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.431029 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.431033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:42.431163 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:42.431213 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 04:35:42.431353 (+ 140us) client_negotiation.cc:240] Negotiation successful
0808 04:35:42.431534 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":161,"thread_start_us":167,"threads_started":1}
I20260808 04:35:42.432617 31402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:42.424726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58870 (local address 127.15.228.67:36541)
0808 04:35:42.425298 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:42.425315 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:42.425356 (+ 41us) server_negotiation.cc:407] Connection header received
0808 04:35:42.425697 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:42.425708 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:42.425979 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:42.426192 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:42.426824 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.428135 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.430041 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.430866 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.431026 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:42.431204 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:42.432340 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0808 04:35:42.432462 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":206,"threads_started":1}
I20260808 04:35:42.432982 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:42.424237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39174 (local address 127.15.228.65:46257)
0808 04:35:42.424674 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:42.424681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:42.425542 (+ 861us) server_negotiation.cc:407] Connection header received
0808 04:35:42.426285 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:42.426293 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:42.426553 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:42.426841 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:42.427220 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.428486 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.430510 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.431428 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.431546 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:42.431696 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:42.432448 (+ 752us) server_negotiation.cc:299] Negotiation successful
0808 04:35:42.432792 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":123,"threads_started":1}
I20260808 04:35:42.433408 31401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:42.424098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:46257 (local address 127.0.0.1:39174)
0808 04:35:42.425398 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:42.425417 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:42.425631 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:42.426912 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:42.426915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:42.426952 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:42.427062 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:42.427070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.428630 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.428633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:42.430274 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:42.430286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.431769 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.431773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:42.432161 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:42.432191 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 04:35:42.432278 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 04:35:42.433255 (+ 977us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":210,"thread_start_us":126,"threads_started":1}
I20260808 04:35:42.435377 31296 tablet_service.cc:1467] Processing CreateTablet for tablet c5d720f0f8744c649ccebbbca75d726a (DEFAULT_TABLE table=test-table [id=eed8034d4f9440a0a7fdf646a58a50b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:42.435617 31082 tablet_service.cc:1467] Processing CreateTablet for tablet c5d720f0f8744c649ccebbbca75d726a (DEFAULT_TABLE table=test-table [id=eed8034d4f9440a0a7fdf646a58a50b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:42.436669 31296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d720f0f8744c649ccebbbca75d726a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:42.436724 31082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d720f0f8744c649ccebbbca75d726a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:42.436455 30902 tablet_service.cc:1467] Processing CreateTablet for tablet c5d720f0f8744c649ccebbbca75d726a (DEFAULT_TABLE table=test-table [id=eed8034d4f9440a0a7fdf646a58a50b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:35:42.437386 30902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d720f0f8744c649ccebbbca75d726a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:35:42.445206 31406 tablet_bootstrap.cc:492] T c5d720f0f8744c649ccebbbca75d726a P 8c89ae6b601b4cb0ab5c06db86c3bae8: Bootstrap starting.
I20260808 04:35:42.445541 31405 tablet_bootstrap.cc:492] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe: Bootstrap starting.
I20260808 04:35:42.447324 31409 tablet_bootstrap.cc:492] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57: Bootstrap starting.
I20260808 04:35:42.447785 31406 tablet_bootstrap.cc:654] T c5d720f0f8744c649ccebbbca75d726a P 8c89ae6b601b4cb0ab5c06db86c3bae8: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:42.448246 31405 tablet_bootstrap.cc:654] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:42.448961 31405 log.cc:824] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:42.449050 31406 log.cc:824] T c5d720f0f8744c649ccebbbca75d726a P 8c89ae6b601b4cb0ab5c06db86c3bae8: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:42.450426 31409 tablet_bootstrap.cc:654] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57: Neither blocks nor log segments found. Creating new log.
I20260808 04:35:42.451980 31406 tablet_bootstrap.cc:492] T c5d720f0f8744c649ccebbbca75d726a P 8c89ae6b601b4cb0ab5c06db86c3bae8: No bootstrap required, opened a new log
I20260808 04:35:42.451980 31405 tablet_bootstrap.cc:492] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe: No bootstrap required, opened a new log
I20260808 04:35:42.452224 31406 ts_tablet_manager.cc:1397] T c5d720f0f8744c649ccebbbca75d726a P 8c89ae6b601b4cb0ab5c06db86c3bae8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:35:42.452224 31405 ts_tablet_manager.cc:1397] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:35:42.452232 31409 log.cc:824] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57: Log is configured to *not* fsync() on all Append() calls
I20260808 04:35:42.454865 31409 tablet_bootstrap.cc:492] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57: No bootstrap required, opened a new log
I20260808 04:35:42.455255 31409 ts_tablet_manager.cc:1397] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20260808 04:35:42.456955 31405 raft_consensus.cc:348] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c89ae6b601b4cb0ab5c06db86c3bae8" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 46257 } } peers { permanent_uuid: "104ccac8503e465cb2941d8765464ebe" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36541 } } peers { permanent_uuid: "306764445d4f42189d6c4299f3bbfb57" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 42255 } }
I20260808 04:35:42.457296 31405 raft_consensus.cc:374] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:42.457170 31406 raft_consensus.cc:348] T c5d720f0f8744c649ccebbbca75d726a P 8c89ae6b601b4cb0ab5c06db86c3bae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c89ae6b601b4cb0ab5c06db86c3bae8" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 46257 } } peers { permanent_uuid: "104ccac8503e465cb2941d8765464ebe" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36541 } } peers { permanent_uuid: "306764445d4f42189d6c4299f3bbfb57" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 42255 } }
I20260808 04:35:42.457351 31405 raft_consensus.cc:729] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 104ccac8503e465cb2941d8765464ebe, State: Initialized, Role: FOLLOWER
I20260808 04:35:42.457439 31406 raft_consensus.cc:374] T c5d720f0f8744c649ccebbbca75d726a P 8c89ae6b601b4cb0ab5c06db86c3bae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:42.457489 31406 raft_consensus.cc:729] T c5d720f0f8744c649ccebbbca75d726a P 8c89ae6b601b4cb0ab5c06db86c3bae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c89ae6b601b4cb0ab5c06db86c3bae8, State: Initialized, Role: FOLLOWER
I20260808 04:35:42.457964 31406 consensus_queue.cc:260] T c5d720f0f8744c649ccebbbca75d726a P 8c89ae6b601b4cb0ab5c06db86c3bae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c89ae6b601b4cb0ab5c06db86c3bae8" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 46257 } } peers { permanent_uuid: "104ccac8503e465cb2941d8765464ebe" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36541 } } peers { permanent_uuid: "306764445d4f42189d6c4299f3bbfb57" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 42255 } }
I20260808 04:35:42.457964 31405 consensus_queue.cc:260] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c89ae6b601b4cb0ab5c06db86c3bae8" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 46257 } } peers { permanent_uuid: "104ccac8503e465cb2941d8765464ebe" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36541 } } peers { permanent_uuid: "306764445d4f42189d6c4299f3bbfb57" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 42255 } }
I20260808 04:35:42.459192 31405 ts_tablet_manager.cc:1428] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260808 04:35:42.459192 31406 ts_tablet_manager.cc:1428] T c5d720f0f8744c649ccebbbca75d726a P 8c89ae6b601b4cb0ab5c06db86c3bae8: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260808 04:35:42.459931 31385 heartbeater.cc:503] Master 127.15.228.126:40429 was elected leader, sending a full tablet report...
I20260808 04:35:42.460073 30992 heartbeater.cc:503] Master 127.15.228.126:40429 was elected leader, sending a full tablet report...
I20260808 04:35:42.459797 31409 raft_consensus.cc:348] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c89ae6b601b4cb0ab5c06db86c3bae8" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 46257 } } peers { permanent_uuid: "104ccac8503e465cb2941d8765464ebe" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36541 } } peers { permanent_uuid: "306764445d4f42189d6c4299f3bbfb57" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 42255 } }
I20260808 04:35:42.460269 31409 raft_consensus.cc:374] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:35:42.460345 31409 raft_consensus.cc:729] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306764445d4f42189d6c4299f3bbfb57, State: Initialized, Role: FOLLOWER
I20260808 04:35:42.460811 31409 consensus_queue.cc:260] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c89ae6b601b4cb0ab5c06db86c3bae8" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 46257 } } peers { permanent_uuid: "104ccac8503e465cb2941d8765464ebe" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36541 } } peers { permanent_uuid: "306764445d4f42189d6c4299f3bbfb57" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 42255 } }
I20260808 04:35:42.461687 31192 heartbeater.cc:503] Master 127.15.228.126:40429 was elected leader, sending a full tablet report...
I20260808 04:35:42.461997 31409 ts_tablet_manager.cc:1428] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
W20260808 04:35:42.533545 30993 tablet.cc:2369] T c5d720f0f8744c649ccebbbca75d726a P 8c89ae6b601b4cb0ab5c06db86c3bae8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:35:42.563910 31193 tablet.cc:2369] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:35:42.611385 31386 tablet.cc:2369] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:35:42.819301 31417 raft_consensus.cc:482] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:35:42.819548 31417 raft_consensus.cc:504] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c89ae6b601b4cb0ab5c06db86c3bae8" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 46257 } } peers { permanent_uuid: "104ccac8503e465cb2941d8765464ebe" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36541 } } peers { permanent_uuid: "306764445d4f42189d6c4299f3bbfb57" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 42255 } }
I20260808 04:35:42.821038 31417 leader_election.cc:288] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c89ae6b601b4cb0ab5c06db86c3bae8 (127.15.228.65:46257), 104ccac8503e465cb2941d8765464ebe (127.15.228.67:36541)
I20260808 04:35:42.825639 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:42.821372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:36541 (local address 127.15.228.66:54249)
0808 04:35:42.821772 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:42.821789 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:42.821993 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:42.822307 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:42.822310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:42.822329 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:42.822502 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:42.822518 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.823487 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.823492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:42.824434 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:42.824445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.825169 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.825174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:42.825289 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:42.825321 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 04:35:42.825401 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 04:35:42.825475 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":97,"threads_started":1}
I20260808 04:35:42.825675 31402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:42.821519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:54249 (local address 127.15.228.67:36541)
0808 04:35:42.821790 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:42.821798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:42.821879 (+ 81us) server_negotiation.cc:407] Connection header received
0808 04:35:42.822130 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:42.822133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:42.822196 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:42.822277 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:42.822655 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.823367 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.824577 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.825037 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.825137 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:42.825182 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:42.825459 (+ 277us) server_negotiation.cc:299] Negotiation successful
0808 04:35:42.825537 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260808 04:35:42.825917 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:42.821640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:46257 (local address 127.15.228.66:34269)
0808 04:35:42.821972 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:42.821984 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:35:42.822136 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:35:42.822450 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:35:42.822452 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:35:42.822468 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:35:42.822581 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:42.822611 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.823618 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.823621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:42.824598 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:35:42.824608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.825470 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.825474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:35:42.825571 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:42.825594 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:35:42.825674 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 04:35:42.825750 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":52,"thread_start_us":64,"threads_started":1}
I20260808 04:35:42.825959 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:42.821758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:34269 (local address 127.15.228.65:46257)
0808 04:35:42.821905 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:42.821911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:42.822112 (+ 201us) server_negotiation.cc:407] Connection header received
0808 04:35:42.822244 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:42.822267 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:42.822330 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:42.822444 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:35:42.822726 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.823468 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.824801 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.825293 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.825430 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:42.825476 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:42.825719 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 04:35:42.825814 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:35:42.826545 30927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d720f0f8744c649ccebbbca75d726a" candidate_uuid: "306764445d4f42189d6c4299f3bbfb57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c89ae6b601b4cb0ab5c06db86c3bae8" is_pre_election: true
I20260808 04:35:42.826599 31322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d720f0f8744c649ccebbbca75d726a" candidate_uuid: "306764445d4f42189d6c4299f3bbfb57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "104ccac8503e465cb2941d8765464ebe" is_pre_election: true
I20260808 04:35:42.826910 30927 raft_consensus.cc:2454] T c5d720f0f8744c649ccebbbca75d726a P 8c89ae6b601b4cb0ab5c06db86c3bae8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 306764445d4f42189d6c4299f3bbfb57 in term 0.
I20260808 04:35:42.827030 31322 raft_consensus.cc:2454] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 306764445d4f42189d6c4299f3bbfb57 in term 0.
I20260808 04:35:42.827579 31025 leader_election.cc:302] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 104ccac8503e465cb2941d8765464ebe, 306764445d4f42189d6c4299f3bbfb57, 8c89ae6b601b4cb0ab5c06db86c3bae8; no voters:
I20260808 04:35:42.827910 31417 raft_consensus.cc:2781] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:35:42.828006 31417 raft_consensus.cc:482] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:35:42.828078 31417 raft_consensus.cc:3037] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:42.829339 31417 raft_consensus.cc:504] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c89ae6b601b4cb0ab5c06db86c3bae8" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 46257 } } peers { permanent_uuid: "104ccac8503e465cb2941d8765464ebe" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36541 } } peers { permanent_uuid: "306764445d4f42189d6c4299f3bbfb57" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 42255 } }
I20260808 04:35:42.829722 31417 leader_election.cc:288] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [CANDIDATE]: Term 1 election: Requested vote from peers 8c89ae6b601b4cb0ab5c06db86c3bae8 (127.15.228.65:46257), 104ccac8503e465cb2941d8765464ebe (127.15.228.67:36541)
I20260808 04:35:42.830154 31322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d720f0f8744c649ccebbbca75d726a" candidate_uuid: "306764445d4f42189d6c4299f3bbfb57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "104ccac8503e465cb2941d8765464ebe"
I20260808 04:35:42.830153 30927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d720f0f8744c649ccebbbca75d726a" candidate_uuid: "306764445d4f42189d6c4299f3bbfb57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c89ae6b601b4cb0ab5c06db86c3bae8"
I20260808 04:35:42.830300 31322 raft_consensus.cc:3037] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:42.830300 30927 raft_consensus.cc:3037] T c5d720f0f8744c649ccebbbca75d726a P 8c89ae6b601b4cb0ab5c06db86c3bae8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:35:42.832114 30927 raft_consensus.cc:2454] T c5d720f0f8744c649ccebbbca75d726a P 8c89ae6b601b4cb0ab5c06db86c3bae8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 306764445d4f42189d6c4299f3bbfb57 in term 1.
I20260808 04:35:42.832114 31322 raft_consensus.cc:2454] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 306764445d4f42189d6c4299f3bbfb57 in term 1.
I20260808 04:35:42.832442 31025 leader_election.cc:302] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 104ccac8503e465cb2941d8765464ebe, 306764445d4f42189d6c4299f3bbfb57; no voters:
I20260808 04:35:42.832669 31417 raft_consensus.cc:2781] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:35:42.832952 31417 raft_consensus.cc:686] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [term 1 LEADER]: Becoming Leader. State: Replica: 306764445d4f42189d6c4299f3bbfb57, State: Running, Role: LEADER
I20260808 04:35:42.833417 31417 consensus_queue.cc:237] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c89ae6b601b4cb0ab5c06db86c3bae8" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 46257 } } peers { permanent_uuid: "104ccac8503e465cb2941d8765464ebe" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36541 } } peers { permanent_uuid: "306764445d4f42189d6c4299f3bbfb57" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 42255 } }
I20260808 04:35:42.836966 30724 catalog_manager.cc:5495] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 306764445d4f42189d6c4299f3bbfb57 (127.15.228.66). New cstate: current_term: 1 leader_uuid: "306764445d4f42189d6c4299f3bbfb57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c89ae6b601b4cb0ab5c06db86c3bae8" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 46257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "104ccac8503e465cb2941d8765464ebe" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "306764445d4f42189d6c4299f3bbfb57" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 42255 } health_report { overall_health: HEALTHY } } }
I20260808 04:35:42.896407 31404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:42.892572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60084 (local address 127.15.228.66:42255)
0808 04:35:42.892735 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 04:35:42.892741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:35:42.892776 (+ 35us) server_negotiation.cc:407] Connection header received
0808 04:35:42.892855 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:35:42.892858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:35:42.892918 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:35:42.893039 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:35:42.893316 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.893891 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.894800 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:35:42.895074 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:35:42.895185 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:35:42.895527 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:35:42.895987 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:35:42.896126 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 04:35:42.896189 (+ 63us) server_negotiation.cc:299] Negotiation successful
0808 04:35:42.896249 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:35:42.903743 30927 raft_consensus.cc:1261] T c5d720f0f8744c649ccebbbca75d726a P 8c89ae6b601b4cb0ab5c06db86c3bae8 [term 1 FOLLOWER]: Refusing update from remote peer 306764445d4f42189d6c4299f3bbfb57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:35:42.903975 31322 raft_consensus.cc:1261] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe [term 1 FOLLOWER]: Refusing update from remote peer 306764445d4f42189d6c4299f3bbfb57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:35:42.904644 31417 consensus_queue.cc:1035] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c89ae6b601b4cb0ab5c06db86c3bae8" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 46257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:35:42.904996 31420 consensus_queue.cc:1035] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "104ccac8503e465cb2941d8765464ebe" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:35:42.913961 31425 mvcc.cc:205] Tried to move back new op lower bound from 7316126690922975232 to 7316126690648813568. Current Snapshot: MvccSnapshot[applied={T|T < 7316126690922975232}]
I20260808 04:35:42.915455 31426 mvcc.cc:205] Tried to move back new op lower bound from 7316126690922975232 to 7316126690648813568. Current Snapshot: MvccSnapshot[applied={T|T < 7316126690922975232}]
I20260808 04:35:42.915436 31428 mvcc.cc:205] Tried to move back new op lower bound from 7316126690922975232 to 7316126690648813568. Current Snapshot: MvccSnapshot[applied={T|T < 7316126690922975232}]
I20260808 04:35:42.925688 30727 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:35:42.925880 30727 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:35:42.933048 30727 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f270d8d34b1442bba3c423c3e206dca: Sending DeleteTablet for 3 replicas of tablet c5d720f0f8744c649ccebbbca75d726a
I20260808 04:35:42.934307 30902 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d720f0f8744c649ccebbbca75d726a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:42 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:39174
I20260808 04:35:42.934859 31296 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d720f0f8744c649ccebbbca75d726a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:42 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:58870
I20260808 04:35:42.935901 31441 tablet_replica.cc:330] T c5d720f0f8744c649ccebbbca75d726a P 8c89ae6b601b4cb0ab5c06db86c3bae8: stopping tablet replica
I20260808 04:35:42.935959 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 30818
I20260808 04:35:42.935540 31082 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d720f0f8744c649ccebbbca75d726a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:35:42 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:60068
I20260808 04:35:42.939440 31443 tablet_replica.cc:330] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57: stopping tablet replica
I20260808 04:35:42.939899 31443 raft_consensus.cc:2229] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:35:42.940526 31443 raft_consensus.cc:2258] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:42.941452 31442 tablet_replica.cc:330] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe: stopping tablet replica
I20260808 04:35:42.942028 31442 raft_consensus.cc:2229] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:35:42.942528 31442 raft_consensus.cc:2258] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:35:42.943727 31443 ts_tablet_manager.cc:1905] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:42.944723 31442 ts_tablet_manager.cc:1905] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:35:42.947530 31443 ts_tablet_manager.cc:1918] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:35:42.947574 31442 ts_tablet_manager.cc:1918] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:35:42.947654 31443 log.cc:1196] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/c5d720f0f8744c649ccebbbca75d726a
I20260808 04:35:42.947677 31442 log.cc:1196] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/c5d720f0f8744c649ccebbbca75d726a
I20260808 04:35:42.948117 31443 ts_tablet_manager.cc:1939] T c5d720f0f8744c649ccebbbca75d726a P 306764445d4f42189d6c4299f3bbfb57: Deleting consensus metadata
I20260808 04:35:42.948244 31442 ts_tablet_manager.cc:1939] T c5d720f0f8744c649ccebbbca75d726a P 104ccac8503e465cb2941d8765464ebe: Deleting consensus metadata
I20260808 04:35:42.949592 30700 catalog_manager.cc:4841] TS 306764445d4f42189d6c4299f3bbfb57 (127.15.228.66:42255): tablet c5d720f0f8744c649ccebbbca75d726a (table test-table [id=eed8034d4f9440a0a7fdf646a58a50b6]) successfully deleted
I20260808 04:35:42.949663 30705 catalog_manager.cc:4841] TS 104ccac8503e465cb2941d8765464ebe (127.15.228.67:36541): tablet c5d720f0f8744c649ccebbbca75d726a (table test-table [id=eed8034d4f9440a0a7fdf646a58a50b6]) successfully deleted
W20260808 04:35:42.951978 30700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.228.65:46257 (error 108)
I20260808 04:35:42.952859 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 30998
I20260808 04:35:42.968528 31445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:35:42.952802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:46257 (local address 127.0.0.1:39180)
0808 04:35:42.968276 (+ 15474us) negotiation.cc:107] Waiting for socket to connect
0808 04:35:42.968406 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.65:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
W20260808 04:35:42.968762 30700 catalog_manager.cc:4584] TS 8c89ae6b601b4cb0ab5c06db86c3bae8 (127.15.228.65:46257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c5d720f0f8744c649ccebbbca75d726a: Network error: Client connection negotiation failed: client connection to 127.15.228.65:46257: connect: Connection refused (error 111)
I20260808 04:35:42.976063 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 31197
I20260808 04:35:42.993475 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 30681
2026-08-08T04:35:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3388 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:35:43 dist-test-slave-drl0 krb5kdc[31448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:35:43 dist-test-slave-drl0 krb5kdc[31448](info): set up 2 sockets
Aug 08 04:35:43 dist-test-slave-drl0 krb5kdc[31448](info): commencing operation
krb5kdc: starting...
W20260808 04:35:44.277905 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:35:44 dist-test-slave-drl0 krb5kdc[31448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:35:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:35:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.228.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.228.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.228.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/rangeradmin_127.15.228.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.228.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/rangeradmin_127.15.228.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.228.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.228.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.228.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/rangerlookup_127.15.228.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.228.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/rangerlookup_127.15.228.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.228.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.228.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.228.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/HTTP_127.15.228.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.228.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/HTTP_127.15.228.84@KRBTEST.COM.keytab.
I20260808 04:35:44.475831 16273 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhFaBKn/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/postgres -l logfile start
2026-08-08 04:35:47.469 UTC [31492] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 04:35:47.469 UTC [31492] LOG: listening on IPv4 address "127.15.228.84", port 43475
2026-08-08 04:35:47.474 UTC [31492] LOG: listening on Unix socket "/tmp/.s.PGSQL.43475"
2026-08-08 04:35:47.518 UTC [31496] LOG: database system was shut down at 2026-08-08 04:35:45 UTC
2026-08-08 04:35:47.524 UTC [31492] LOG: database system is ready to accept connections
I20260808 04:35:48.738997 16273 mini_postgres.cc:96] Postgres bound to 43475
2026-08-08 04:35:48.748 UTC [31505] FATAL: database "slave" does not exist
127.15.228.84:43475 - accepting connections
I20260808 04:35:48.749630 16273 mini_ranger.cc:161] Starting Ranger...
I20260808 04:35:48.769821 16273 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 04:35:49.400970 16273 mini_ranger.cc:87] Created ranger Postgres database
I20260808 04:35:49.401119 16273 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ranger-admin
2026-08-08 04:35:49,735 [I] DB FLAVOR :POSTGRES
2026-08-08 04:35:49,735 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 04:35:49,735 [I] Checking connection..
2026-08-08 04:35:49,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 04:35:50,033 [I] Checking connection passed.
2026-08-08 04:35:50,034 [I] --------- Verifying version history table ---------
2026-08-08 04:35:50,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:35:50,340 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 04:35:50,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:35:50,635 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 04:35:50,635 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 04:35:50,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 04:35:50.919 UTC [31604] WARNING: there is no transaction in progress
2026-08-08 04:35:50,928 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 04:35:50,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:35:51,228 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 04:35:51,228 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 04:35:51,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 04:35:51,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 04:35:51,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-drl0.c.gcp-upstream.internal','N') ;"
2026-08-08 04:35:52,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 04:35:52,322 [I] Table x_portal_user does not exist in database ranger
2026-08-08 04:35:52,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 04:35:52,611 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 04:35:52,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 04:35:52,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 04:35:53,153 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 04:35:53,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 04:35:54.321 UTC [31750] WARNING: there is no transaction in progress
2026-08-08 04:35:54.341 UTC [31750] WARNING: there is no transaction in progress
2026-08-08 04:35:54.360 UTC [31750] WARNING: there is no transaction in progress
2026-08-08 04:35:54.379 UTC [31750] WARNING: there is no transaction in progress
2026-08-08 04:35:54.399 UTC [31750] WARNING: there is no transaction in progress
2026-08-08 04:35:54.419 UTC [31750] WARNING: there is no transaction in progress
2026-08-08 04:35:54.483 UTC [31750] WARNING: there is no transaction in progress
2026-08-08 04:35:54.491 UTC [31750] WARNING: there is no transaction in progress
2026-08-08 04:35:54.510 UTC [31750] WARNING: there is no transaction in progress
2026-08-08 04:35:54.521 UTC [31750] WARNING: there is no transaction in progress
2026-08-08 04:35:54.533 UTC [31750] WARNING: there is no transaction in progress
2026-08-08 04:35:54.553 UTC [31750] WARNING: there is no transaction in progress
2026-08-08 04:35:54.566 UTC [31750] WARNING: there is no transaction in progress
2026-08-08 04:35:54.574 UTC [31750] WARNING: there is no transaction in progress
2026-08-08 04:35:54.584 UTC [31750] WARNING: there is no transaction in progress
2026-08-08 04:35:54.593 UTC [31750] WARNING: there is no transaction in progress
2026-08-08 04:35:55,463 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 04:35:55,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 04:35:55,775 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 04:35:55,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 04:35:56,094 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 04:35:56,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 04:35:56,397 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 04:35:56,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 04:35:56,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 04:35:56,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-drl0.c.gcp-upstream.internal';"
2026-08-08 04:35:57,275 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 04:35:57,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 04:35:57,573 [I] DB_PATCHES have already been applied
I20260808 04:35:57.580121 16273 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskhFaBKn/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 04:35:57.580227 16273 mini_ranger.cc:193] Using host: 127.15.228.84
I20260808 04:35:57.586875 16273 mini_ranger.cc:227] Ranger admin URL: http://127.15.228.84:55369
Aug 08, 2026 4:35:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 4:35:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 4:35:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 4:35:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 4:35:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 4:35:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 4:35:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 04:35:59.014961 16273 mini_ranger.cc:160] Time spent starting Ranger: real 10.265s user 0.000s sys 0.018s
Aug 08, 2026 4:36:24 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 4:36:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 08, 2026 4:36:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 4:36:24 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 4:36:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 4:36:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 4:36:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 4:36:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 4:36:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 4:36:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 4:36:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 4:36:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 4:36:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 4:36:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 4:36:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 4:36:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 4:36:24 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 4:36:25 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 04:36:25.963765 16273 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.228.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.228.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.228.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/rangerkms_127.15.228.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.228.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/rangerkms_127.15.228.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.228.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.228.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.228.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/HTTP_127.15.228.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.228.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/HTTP_127.15.228.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 04:36:26 dist-test-slave-drl0 krb5kdc[31448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163786, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 04:36:26.072747 16273 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 04:36:26.091073 16273 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 04:36:27.099331 16273 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 04:36:27.099462 16273 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ranger-kms
I20260808 04:36:27.099589 16273 mini_ranger_kms.cc:227] Using postgres at 127.15.228.84:43475
2026-08-08 04:36:27,909 [I] DB FLAVOR :POSTGRES
2026-08-08 04:36:27,909 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 04:36:27,909 [I] Checking connection
2026-08-08 04:36:27,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 04:36:28,255 [I] connection success
2026-08-08 04:36:28,255 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 04:36:28,255 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 04:36:28,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 04:36:28,567 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 04:36:28,568 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 04:36:28,568 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 04:36:28,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:43475/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 04:36:28,904 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 04:36:29.579392 16273 mini_ranger_kms.cc:310] Created kms service
I20260808 04:36:29.803054 16273 mini_ranger_kms.cc:326] Created kudu user
I20260808 04:36:29.856375 16273 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 04:36:29 dist-test-slave-drl0 krb5kdc[31448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163786, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
I20260808 04:36:30.344491 16273 mini_ranger_kms.cc:384] Added ranger policy
I20260808 04:36:30.344671 16273 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 04:36:30.344697 16273 mini_ranger_kms.cc:240] Using host: 127.15.228.84
I20260808 04:36:30.350889 16273 mini_ranger_kms.cc:276] Ranger KMS PID: 32007
I20260808 04:36:30.351106 16273 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.228.84:46801
Aug 08, 2026 4:36:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 4:36:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 4:36:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 4:36:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 4:36:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 4:36:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260808 04:36:33.180538 16273 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 7.108s user 0.013s sys 0.525s
I20260808 04:36:33.180697 16273 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 04:36:33.180855 16273 mini_ranger_kms.cc:401] 127.15.228.84:46801/kms/v1/keys
Aug 08 04:36:33 dist-test-slave-drl0 krb5kdc[31448](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786163793, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.228.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 08 04:36:33 dist-test-slave-drl0 krb5kdc[31448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163793, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.228.84@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
Aug 08 04:36:34 dist-test-slave-drl0 krb5kdc[31448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163793, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.228.84@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
Aug 08 04:36:36 dist-test-slave-drl0 krb5kdc[31448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:36:36.990821 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:43681
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:43681
--ranger_config_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.228.84:46801/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:36:37.140057 32075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:36:37.140637 32075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:36:37.141028 32075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:36:37.147624 32075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:36:37.147758 32075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:36:37.147781 32075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:36:37.147812 32075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:36:37.147830 32075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:36:37.157335 32075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.228.84:46801/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:43681
--ranger_config_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:43681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:36:37.158921 32075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:36:37.160200 32075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:36:37.172685 32083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:36:37.172695 32085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:36:37.172605 32082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:36:37.172605 32075 server_base.cc:1034] running on GCE node
I20260808 04:36:37.173869 32075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:36:37.175561 32075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:36:37.176810 32075 hybrid_clock.cc:648] HybridClock initialized: now 1786163797176752 us; error 77 us; skew 500 ppm
Aug 08 04:36:37 dist-test-slave-drl0 krb5kdc[31448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163797, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:36:37.184338 32075 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:36:37.185520 32075 webserver.cc:460] Webserver started at http://127.15.228.126:46881/ using document root <none> and password file <none>
I20260808 04:36:37.186172 32075 fs_manager.cc:359] Metadata directory not provided
I20260808 04:36:37.186249 32075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:36:37.186506 32075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:36:37 dist-test-slave-drl0 krb5kdc[31448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163797, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
I20260808 04:36:37.267426 32075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "4fe36bd6e08c4a2289750d21a01b0630"
format_stamp: "Formatted at 2026-08-08 04:36:37 on dist-test-slave-drl0"
server_key: "dde6c44841f15761e066410515d5060b"
server_key_iv: "cb7508729c780d158ff8ab0310dbf1f3"
server_key_version: "kuduclusterkey@0"
I20260808 04:36:37.268273 32075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "4fe36bd6e08c4a2289750d21a01b0630"
format_stamp: "Formatted at 2026-08-08 04:36:37 on dist-test-slave-drl0"
server_key: "dde6c44841f15761e066410515d5060b"
server_key_iv: "cb7508729c780d158ff8ab0310dbf1f3"
server_key_version: "kuduclusterkey@0"
I20260808 04:36:37.273811 32075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:36:37.296852 32095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:36:37.299098 32075 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:36:37.299372 32075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "4fe36bd6e08c4a2289750d21a01b0630"
format_stamp: "Formatted at 2026-08-08 04:36:37 on dist-test-slave-drl0"
server_key: "dde6c44841f15761e066410515d5060b"
server_key_iv: "cb7508729c780d158ff8ab0310dbf1f3"
server_key_version: "kuduclusterkey@0"
I20260808 04:36:37.299580 32075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:36:37.312635 32075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:36:37.317380 32075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:36:37.318073 32075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:36:37.328869 32075 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:43681
I20260808 04:36:37.328980 32166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:43681 every 8 connection(s)
I20260808 04:36:37.330413 32075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:36:37.332957 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 32075
I20260808 04:36:37.333096 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:36:37.333400 16273 external_mini_cluster.cc:1428] Setting key f7ccee626bdb7d4bca4c6b2f3fff2c21
I20260808 04:36:37.335397 32167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:36:37.342495 32167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630: Bootstrap starting.
Aug 08 04:36:37 dist-test-slave-drl0 krb5kdc[31448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:36:37.345710 32167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630: Neither blocks nor log segments found. Creating new log.
I20260808 04:36:37.346725 32167 log.cc:824] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630: Log is configured to *not* fsync() on all Append() calls
I20260808 04:36:37.349596 32167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630: No bootstrap required, opened a new log
I20260808 04:36:37.354238 32170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:37.335235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.15.228.126:43681)
0808 04:36:37.335941 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0808 04:36:37.335969 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0808 04:36:37.336016 (+ 47us) server_negotiation.cc:407] Connection header received
0808 04:36:37.337025 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:36:37.337049 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:36:37.337473 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:36:37.337842 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:36:37.338872 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:37.340570 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:37.341815 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:37.342533 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:37.342696 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:36:37.346222 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:36:37.346230 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:36:37.346258 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:36:37.346295 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:36:37.349191 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:36:37.349767 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:36:37.349773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:36:37.349780 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:36:37.350179 (+ 399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:36:37.350700 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:36:37.350704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:36:37.350706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:36:37.351568 (+ 862us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:36:37.351885 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0808 04:36:37.352428 (+ 543us) server_negotiation.cc:299] Negotiation successful
0808 04:36:37.352850 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":160,"threads_started":1}
I20260808 04:36:37.355477 32167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe36bd6e08c4a2289750d21a01b0630" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43681 } }
I20260808 04:36:37.355777 32167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:36:37.355857 32167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe36bd6e08c4a2289750d21a01b0630, State: Initialized, Role: FOLLOWER
I20260808 04:36:37.356395 32167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe36bd6e08c4a2289750d21a01b0630" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43681 } }
I20260808 04:36:37.356568 32167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:36:37.356668 32167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:36:37.356791 32167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:36:37.358232 32167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe36bd6e08c4a2289750d21a01b0630" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43681 } }
I20260808 04:36:37.358601 32167 leader_election.cc:302] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fe36bd6e08c4a2289750d21a01b0630; no voters:
I20260808 04:36:37.358991 32167 leader_election.cc:288] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:36:37.359088 32172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:36:37.359354 32172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630 [term 1 LEADER]: Becoming Leader. State: Replica: 4fe36bd6e08c4a2289750d21a01b0630, State: Running, Role: LEADER
I20260808 04:36:37.359728 32172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe36bd6e08c4a2289750d21a01b0630" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43681 } }
I20260808 04:36:37.360424 32167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:36:37.361052 32172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fe36bd6e08c4a2289750d21a01b0630. Latest consensus state: current_term: 1 leader_uuid: "4fe36bd6e08c4a2289750d21a01b0630" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe36bd6e08c4a2289750d21a01b0630" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43681 } } }
I20260808 04:36:37.361186 32172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630 [sys.catalog]: This master's current role is: LEADER
I20260808 04:36:37.362131 32174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fe36bd6e08c4a2289750d21a01b0630" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe36bd6e08c4a2289750d21a01b0630" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43681 } } }
I20260808 04:36:37.362327 32174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630 [sys.catalog]: This master's current role is: LEADER
I20260808 04:36:37.362198 32181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:36:37.366263 32181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:36:37.367054 32167 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 04:36:37.387225 32181 catalog_manager.cc:1347] Generated new cluster ID: 25140f138e45418e88f1560764c2b4ef
I20260808 04:36:37.387343 32181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:36:37.411697 32181 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:36:37.415189 32181 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:36:37.442150 32181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630: Generated new TSK 0
I20260808 04:36:37.443430 32181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
I20260808 04:36:38.359781 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.228.84:46801/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:43681
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:36:38.569731 32227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:36:38.570245 32227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:36:38.570397 32227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:36:38.579173 32227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:36:38.579317 32227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:36:38.579443 32227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:36:38.586243 32227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.228.84:46801/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:43681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:36:38.587952 32227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:36:38.589715 32227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:36:38.609790 32239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:36:38.610270 32241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:36:38.610517 32238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:36:38.610967 32227 server_base.cc:1034] running on GCE node
I20260808 04:36:38.611480 32227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:36:38.612360 32227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:36:38.613706 32227 hybrid_clock.cc:648] HybridClock initialized: now 1786163798613671 us; error 55 us; skew 500 ppm
Aug 08 04:36:38 dist-test-slave-drl0 krb5kdc[31448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163798, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:36:38.617437 32227 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:36:38.618551 32227 webserver.cc:460] Webserver started at http://127.15.228.65:38119/ using document root <none> and password file <none>
I20260808 04:36:38.619248 32227 fs_manager.cc:359] Metadata directory not provided
I20260808 04:36:38.619347 32227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:36:38.619599 32227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:36:38 dist-test-slave-drl0 krb5kdc[31448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163798, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
I20260808 04:36:38.654366 32227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "bb13c0eed3064189b5ec30b86987761f"
format_stamp: "Formatted at 2026-08-08 04:36:38 on dist-test-slave-drl0"
server_key: "bc33800dce3a5708085591016a50f1ed"
server_key_iv: "7027656f8ba946bd95a657dfefdf74f7"
server_key_version: "kuduclusterkey@0"
I20260808 04:36:38.656217 32227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "bb13c0eed3064189b5ec30b86987761f"
format_stamp: "Formatted at 2026-08-08 04:36:38 on dist-test-slave-drl0"
server_key: "bc33800dce3a5708085591016a50f1ed"
server_key_iv: "7027656f8ba946bd95a657dfefdf74f7"
server_key_version: "kuduclusterkey@0"
I20260808 04:36:38.663192 32227 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260808 04:36:38.695820 32254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:36:38.698102 32227 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.001s
I20260808 04:36:38.698426 32227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "bb13c0eed3064189b5ec30b86987761f"
format_stamp: "Formatted at 2026-08-08 04:36:38 on dist-test-slave-drl0"
server_key: "bc33800dce3a5708085591016a50f1ed"
server_key_iv: "7027656f8ba946bd95a657dfefdf74f7"
server_key_version: "kuduclusterkey@0"
I20260808 04:36:38.698706 32227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:36:38.745039 32227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:36:38.753938 32227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:36:38.754309 32227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:36:38.755510 32227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:36:38.757151 32227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:36:38.757376 32227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 04:36:38.757586 32227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:36:38.757702 32227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:36:38.783146 32415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:34527 every 8 connection(s)
I20260808 04:36:38.782886 32227 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:34527
I20260808 04:36:38.784579 32227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:36:38.785832 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 32227
I20260808 04:36:38.785962 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:36:38.786300 16273 external_mini_cluster.cc:1428] Setting key 9619aa27e4107d22227fbb2b407adbc7
Aug 08 04:36:38 dist-test-slave-drl0 krb5kdc[31448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163798, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.228.66@KRBTEST.COM; defaulting to no policy
I20260808 04:36:38.808542 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:38.787303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:45975 (local address 127.15.228.126:43681)
0808 04:36:38.787732 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0808 04:36:38.787738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:36:38.791670 (+ 3932us) server_negotiation.cc:407] Connection header received
0808 04:36:38.792682 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:36:38.792686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:36:38.792768 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:36:38.792876 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:36:38.795547 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:38.796692 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:38.798980 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:38.799464 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:38.799648 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:36:38.803466 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:36:38.803473 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:36:38.803476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:36:38.803515 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:36:38.805615 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:36:38.806436 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:36:38.806442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:36:38.806445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:36:38.806563 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:36:38.807061 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:36:38.807068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:36:38.807073 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:36:38.807343 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:36:38.807474 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 04:36:38.808100 (+ 626us) server_negotiation.cc:299] Negotiation successful
0808 04:36:38.808310 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":158,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.66@KRBTEST.COM" created.
I20260808 04:36:38.811193 32422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:38.789204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:43681 (local address 127.15.228.65:45975)
0808 04:36:38.791426 (+ 2222us) negotiation.cc:107] Waiting for socket to connect
0808 04:36:38.791528 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0808 04:36:38.792457 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:36:38.793326 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:36:38.793337 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:36:38.793869 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:36:38.795308 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:36:38.795338 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:36:38.796910 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:36:38.796917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:36:38.798762 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:36:38.798782 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:36:38.799928 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:36:38.799937 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:36:38.800214 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:36:38.801097 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:36:38.801123 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:36:38.803194 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:36:38.805779 (+ 2585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:36:38.805788 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:36:38.805811 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:36:38.806142 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:36:38.806689 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:36:38.806693 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:36:38.806696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:36:38.806892 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:36:38.807474 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:36:38.807483 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:36:38.807814 (+ 331us) client_negotiation.cc:769] Sending connection context
0808 04:36:38.808034 (+ 220us) client_negotiation.cc:240] Negotiation successful
0808 04:36:38.809838 (+ 1804us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1865,"thread_start_us":245,"threads_started":1}
I20260808 04:36:38.813086 32416 heartbeater.cc:347] Connected to a master server at 127.15.228.126:43681
I20260808 04:36:38.813565 32416 heartbeater.cc:464] Registering TS with master...
I20260808 04:36:38.814558 32416 heartbeater.cc:511] Master 127.15.228.126:43681 requested a full tablet report, sending...
I20260808 04:36:38.817623 32117 ts_manager.cc:194] Registered new tserver with Master: bb13c0eed3064189b5ec30b86987761f (127.15.228.65:34527)
I20260808 04:36:38.819988 32117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:45975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
I20260808 04:36:38.841295 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.228.84:46801/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:43681
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:36:38 dist-test-slave-drl0 krb5kdc[31448](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786163798, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 04:36:39.045341 32425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:36:39.045665 32425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:36:39.045766 32425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:36:39.050508 32425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:36:39.050606 32425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:36:39.050701 32425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:36:39.056995 32425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.228.84:46801/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:43681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:36:39.058290 32425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:36:39.059533 32425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:36:39.071933 32434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:36:39.072197 32425 server_base.cc:1034] running on GCE node
W20260808 04:36:39.072156 32433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:36:39.072041 32438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:36:39.073040 32425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:36:39.073899 32425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:36:39.075390 32425 hybrid_clock.cc:648] HybridClock initialized: now 1786163799075367 us; error 67 us; skew 500 ppm
Aug 08 04:36:39 dist-test-slave-drl0 krb5kdc[31448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163799, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:36:39.079617 32425 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:36:39.081063 32425 webserver.cc:460] Webserver started at http://127.15.228.66:39953/ using document root <none> and password file <none>
I20260808 04:36:39.081827 32425 fs_manager.cc:359] Metadata directory not provided
I20260808 04:36:39.081955 32425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:36:39.082216 32425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:36:39 dist-test-slave-drl0 krb5kdc[31448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163799, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
I20260808 04:36:39.113170 32425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "616445a698fd48509f0ab763205ff942"
format_stamp: "Formatted at 2026-08-08 04:36:39 on dist-test-slave-drl0"
server_key: "97dabf7966431ed38c1ec314485ecb2a"
server_key_iv: "82547d43baf45f6834df45d4a6db117b"
server_key_version: "kuduclusterkey@0"
I20260808 04:36:39.114190 32425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "616445a698fd48509f0ab763205ff942"
format_stamp: "Formatted at 2026-08-08 04:36:39 on dist-test-slave-drl0"
server_key: "97dabf7966431ed38c1ec314485ecb2a"
server_key_iv: "82547d43baf45f6834df45d4a6db117b"
server_key_version: "kuduclusterkey@0"
I20260808 04:36:39.119666 32425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:36:39.149147 32451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:36:39.150810 32425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:36:39.151008 32425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "616445a698fd48509f0ab763205ff942"
format_stamp: "Formatted at 2026-08-08 04:36:39 on dist-test-slave-drl0"
server_key: "97dabf7966431ed38c1ec314485ecb2a"
server_key_iv: "82547d43baf45f6834df45d4a6db117b"
server_key_version: "kuduclusterkey@0"
I20260808 04:36:39.151185 32425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:36:39.176577 32425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:36:39.180172 32425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:36:39.180406 32425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:36:39.181432 32425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:36:39.182489 32425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:36:39.182564 32425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:36:39.182633 32425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:36:39.182684 32425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:36:39.197359 32425 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:32947
I20260808 04:36:39.197389 32605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:32947 every 8 connection(s)
I20260808 04:36:39.198800 32425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:36:39.205305 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 32425
I20260808 04:36:39.205502 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:36:39.206122 16273 external_mini_cluster.cc:1428] Setting key bdf095534c6934f9a634e93e6274e100
Aug 08 04:36:39 dist-test-slave-drl0 krb5kdc[31448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163799, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.228.67@KRBTEST.COM; defaulting to no policy
I20260808 04:36:39.223522 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:39.203102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:42927 (local address 127.15.228.126:43681)
0808 04:36:39.203266 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 04:36:39.203272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:36:39.203293 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:36:39.204315 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:36:39.204319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:36:39.204412 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:36:39.204505 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:36:39.208608 (+ 4103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.209739 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.212580 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.213056 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.213171 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:36:39.217078 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:36:39.217087 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:36:39.217091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:36:39.217136 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:36:39.219990 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:36:39.220870 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:36:39.220877 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:36:39.220878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:36:39.221034 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:36:39.221570 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:36:39.221577 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:36:39.221582 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:36:39.221940 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:36:39.222156 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0808 04:36:39.223119 (+ 963us) server_negotiation.cc:299] Negotiation successful
0808 04:36:39.223305 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 04:36:39.224813 32611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:39.202038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:43681 (local address 127.15.228.66:42927)
0808 04:36:39.202684 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0808 04:36:39.202756 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0808 04:36:39.203969 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:36:39.204948 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:36:39.204962 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:36:39.206007 (+ 1045us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:36:39.208353 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:36:39.208390 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.210446 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.210453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:36:39.212382 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:36:39.212397 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.213317 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.213325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:36:39.213615 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:36:39.214491 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:36:39.214523 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:36:39.216793 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:36:39.220205 (+ 3412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:36:39.220216 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:36:39.220242 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:36:39.220710 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:36:39.221172 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:36:39.221178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:36:39.221184 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:36:39.221422 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:36:39.222244 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:36:39.222252 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:36:39.222790 (+ 538us) client_negotiation.cc:769] Sending connection context
0808 04:36:39.223055 (+ 265us) client_negotiation.cc:240] Negotiation successful
0808 04:36:39.223522 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":182,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.67@KRBTEST.COM" created.
I20260808 04:36:39.226621 32606 heartbeater.cc:347] Connected to a master server at 127.15.228.126:43681
I20260808 04:36:39.226912 32606 heartbeater.cc:464] Registering TS with master...
I20260808 04:36:39.227746 32606 heartbeater.cc:511] Master 127.15.228.126:43681 requested a full tablet report, sending...
I20260808 04:36:39.229128 32117 ts_manager.cc:194] Registered new tserver with Master: 616445a698fd48509f0ab763205ff942 (127.15.228.66:32947)
I20260808 04:36:39.230167 32117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:42927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
I20260808 04:36:39.257450 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.228.84:46801/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:43681
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:36:39.430133 32616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:36:39.430653 32616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:36:39.430898 32616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:36:39.435964 32616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:36:39.436064 32616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:36:39.436179 32616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:36:39.443645 32616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.228.84:46801/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:43681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:36:39.445222 32616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:36:39.446539 32616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:36:39.460024 32623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:36:39.460340 32622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:36:39.460011 32625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:36:39.462033 32616 server_base.cc:1034] running on GCE node
I20260808 04:36:39.462607 32616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:36:39.463521 32616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:36:39.464813 32616 hybrid_clock.cc:648] HybridClock initialized: now 1786163799464781 us; error 43 us; skew 500 ppm
Aug 08 04:36:39 dist-test-slave-drl0 krb5kdc[31448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163799, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:36:39.469249 32616 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:36:39.470270 32616 webserver.cc:460] Webserver started at http://127.15.228.67:43705/ using document root <none> and password file <none>
I20260808 04:36:39.471071 32616 fs_manager.cc:359] Metadata directory not provided
I20260808 04:36:39.471177 32616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:36:39.471449 32616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:36:39 dist-test-slave-drl0 krb5kdc[31448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163799, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
I20260808 04:36:39.497605 32616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "73f5eaca8b374bba8b75f00e8c954e42"
format_stamp: "Formatted at 2026-08-08 04:36:39 on dist-test-slave-drl0"
server_key: "a8080d0367e37cb47366bb6083c3a225"
server_key_iv: "3eb06010552e182142df40f2f72e25a8"
server_key_version: "kuduclusterkey@0"
I20260808 04:36:39.510350 32616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "73f5eaca8b374bba8b75f00e8c954e42"
format_stamp: "Formatted at 2026-08-08 04:36:39 on dist-test-slave-drl0"
server_key: "a8080d0367e37cb47366bb6083c3a225"
server_key_iv: "3eb06010552e182142df40f2f72e25a8"
server_key_version: "kuduclusterkey@0"
I20260808 04:36:39.521416 32616 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.006s sys 0.000s
Aug 08 04:36:39 dist-test-slave-drl0 krb5kdc[31448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163798, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
I20260808 04:36:39.559705 32635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:36:39.562248 32616 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.004s sys 0.001s
I20260808 04:36:39.562631 32616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "73f5eaca8b374bba8b75f00e8c954e42"
format_stamp: "Formatted at 2026-08-08 04:36:39 on dist-test-slave-drl0"
server_key: "a8080d0367e37cb47366bb6083c3a225"
server_key_iv: "3eb06010552e182142df40f2f72e25a8"
server_key_version: "kuduclusterkey@0"
I20260808 04:36:39.562839 32616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:36:39.594008 32616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:36:39.599076 32616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:36:39.599378 32616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:36:39.600263 32616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:36:39.601558 32616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:36:39.601675 32616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:36:39.601787 32616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:36:39.601858 32616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:36:39.622237 32616 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:42425
I20260808 04:36:39.623723 32616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:36:39.625494 331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:42425 every 8 connection(s)
I20260808 04:36:39.626485 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 32616
I20260808 04:36:39.626624 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:36:39.626987 16273 external_mini_cluster.cc:1428] Setting key 822227294dc9569e594c914aa9e9880f
Aug 08 04:36:39 dist-test-slave-drl0 krb5kdc[31448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163799, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:36:39.648509 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:39.629991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:46547 (local address 127.15.228.126:43681)
0808 04:36:39.630135 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 04:36:39.630140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:36:39.631204 (+ 1064us) server_negotiation.cc:407] Connection header received
0808 04:36:39.633261 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:36:39.633266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:36:39.633528 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:36:39.633610 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:36:39.635691 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.636730 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.638745 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.639141 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.639228 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:36:39.642605 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:36:39.642610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:36:39.642613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:36:39.642641 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:36:39.644569 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:36:39.645263 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:36:39.645267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:36:39.645269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:36:39.645356 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:36:39.645787 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:36:39.645791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:36:39.645792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:36:39.646039 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:36:39.646110 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 04:36:39.648036 (+ 1926us) server_negotiation.cc:299] Negotiation successful
0808 04:36:39.648274 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:36:39.651340 334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:39.628787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:43681 (local address 127.15.228.67:46547)
0808 04:36:39.631021 (+ 2234us) negotiation.cc:107] Waiting for socket to connect
0808 04:36:39.631090 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 04:36:39.632853 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:36:39.634176 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:36:39.634188 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:36:39.634650 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:36:39.635474 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:36:39.635498 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.636934 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.636937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:36:39.638471 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:36:39.638485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.639486 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.639489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:36:39.639711 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:36:39.640502 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:36:39.640525 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:36:39.642391 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:36:39.644730 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:36:39.644736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:36:39.644757 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:36:39.645138 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:36:39.645482 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:36:39.645485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:36:39.645487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:36:39.645678 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:36:39.646195 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:36:39.646202 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:36:39.647207 (+ 1005us) client_negotiation.cc:769] Sending connection context
0808 04:36:39.649455 (+ 2248us) client_negotiation.cc:240] Negotiation successful
0808 04:36:39.649913 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1836,"thread_start_us":182,"threads_started":1}
I20260808 04:36:39.653182 332 heartbeater.cc:347] Connected to a master server at 127.15.228.126:43681
I20260808 04:36:39.653466 332 heartbeater.cc:464] Registering TS with master...
I20260808 04:36:39.654088 332 heartbeater.cc:511] Master 127.15.228.126:43681 requested a full tablet report, sending...
I20260808 04:36:39.656651 32116 ts_manager.cc:194] Registered new tserver with Master: 73f5eaca8b374bba8b75f00e8c954e42 (127.15.228.67:42425)
I20260808 04:36:39.657636 32116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:46547
I20260808 04:36:39.665436 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:36:39.684995 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:39.668421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.15.228.126:43681)
0808 04:36:39.668560 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 04:36:39.668564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:36:39.671508 (+ 2944us) server_negotiation.cc:407] Connection header received
0808 04:36:39.671845 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:36:39.671849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:36:39.671917 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:36:39.671978 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:36:39.673304 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.674235 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.679569 (+ 5334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.679835 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.679994 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:36:39.681596 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:36:39.681600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:36:39.681602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:36:39.681630 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:36:39.683147 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:36:39.683611 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:36:39.683620 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:36:39.683622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:36:39.683699 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:36:39.683980 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:36:39.683987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:36:39.683989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:36:39.684204 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:36:39.684349 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 04:36:39.684658 (+ 309us) server_negotiation.cc:299] Negotiation successful
0808 04:36:39.684796 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:36:39.688907 32116 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:36:39.694474 32116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:36:39.719578 347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:39.708630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:32947 (local address 127.0.0.1:35846)
0808 04:36:39.709073 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0808 04:36:39.709123 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 04:36:39.709255 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:36:39.710764 (+ 1509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:36:39.710772 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:36:39.710804 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:36:39.711054 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:36:39.711062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.714563 (+ 3501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.714568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:36:39.716924 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:36:39.716939 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.717959 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.717963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:36:39.718079 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:36:39.718119 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 04:36:39.718243 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 04:36:39.718393 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":158,"threads_started":1}
I20260808 04:36:39.720323 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:39.709562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35846 (local address 127.15.228.66:32947)
0808 04:36:39.710023 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0808 04:36:39.710051 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0808 04:36:39.710092 (+ 41us) server_negotiation.cc:407] Connection header received
0808 04:36:39.710219 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:36:39.710226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:36:39.710514 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:36:39.711251 (+ 737us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:36:39.711416 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.714393 (+ 2977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.717171 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.717790 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.717941 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:36:39.718120 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 04:36:39.720013 (+ 1893us) server_negotiation.cc:299] Negotiation successful
0808 04:36:39.720157 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":183,"threads_started":1}
I20260808 04:36:39.724390 32513 tablet_service.cc:1467] Processing CreateTablet for tablet 221cea23e1984729b30fa1a70ea2f23b (DEFAULT_TABLE table=test-table [id=8e33d4b8a85d40be86f520ce42663a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:36:39.725737 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:39.711851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:42425 (local address 127.0.0.1:56920)
0808 04:36:39.718606 (+ 6755us) negotiation.cc:107] Waiting for socket to connect
0808 04:36:39.718624 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:36:39.718741 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:36:39.719972 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:36:39.719976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:36:39.719996 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:36:39.720097 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:36:39.720102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.721702 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.721706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:36:39.723843 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:36:39.723854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.725286 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.725290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:36:39.725388 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:36:39.725435 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 04:36:39.725501 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 04:36:39.725577 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6684,"mutex_wait_us":939,"thread_start_us":163,"threads_started":1}
I20260808 04:36:39.726027 32513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221cea23e1984729b30fa1a70ea2f23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:36:39.726588 350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:39.712063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.15.228.67:42425)
0808 04:36:39.718431 (+ 6368us) server_negotiation.cc:206] Beginning negotiation
0808 04:36:39.718447 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:36:39.718845 (+ 398us) server_negotiation.cc:407] Connection header received
0808 04:36:39.718967 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:36:39.718974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:36:39.719236 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:36:39.719423 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:36:39.720232 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.721570 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.724007 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.724863 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.724973 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:36:39.725127 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 04:36:39.726057 (+ 930us) server_negotiation.cc:299] Negotiation successful
0808 04:36:39.726229 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6208,"thread_start_us":121,"threads_started":1}
I20260808 04:36:39.727651 349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:39.713198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:34527 (local address 127.0.0.1:36348)
0808 04:36:39.721395 (+ 8197us) negotiation.cc:107] Waiting for socket to connect
0808 04:36:39.721412 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:36:39.721503 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:36:39.722450 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:36:39.722453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:36:39.722470 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:36:39.722841 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:36:39.722850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.724217 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.724220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:36:39.725328 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:36:39.725336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.727155 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.727159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:36:39.727299 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:36:39.727328 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 04:36:39.727414 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 04:36:39.727494 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6433,"thread_start_us":593,"threads_started":1}
I20260808 04:36:39.728756 352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:39.713384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36348 (local address 127.15.228.65:34527)
0808 04:36:39.721803 (+ 8419us) server_negotiation.cc:206] Beginning negotiation
0808 04:36:39.721811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:36:39.721836 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:36:39.721927 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:36:39.721931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:36:39.722144 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:36:39.722295 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:36:39.723034 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.724094 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.725463 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.726306 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.726615 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:36:39.726811 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 04:36:39.728421 (+ 1610us) server_negotiation.cc:299] Negotiation successful
0808 04:36:39.728579 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8286,"thread_start_us":137,"threads_started":1}
I20260808 04:36:39.731916 32312 tablet_service.cc:1467] Processing CreateTablet for tablet 221cea23e1984729b30fa1a70ea2f23b (DEFAULT_TABLE table=test-table [id=8e33d4b8a85d40be86f520ce42663a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:36:39.731916 32698 tablet_service.cc:1467] Processing CreateTablet for tablet 221cea23e1984729b30fa1a70ea2f23b (DEFAULT_TABLE table=test-table [id=8e33d4b8a85d40be86f520ce42663a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:36:39.733145 32312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221cea23e1984729b30fa1a70ea2f23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:36:39.733146 32698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221cea23e1984729b30fa1a70ea2f23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:36:39.740880 355 tablet_bootstrap.cc:492] T 221cea23e1984729b30fa1a70ea2f23b P 616445a698fd48509f0ab763205ff942: Bootstrap starting.
I20260808 04:36:39.743584 355 tablet_bootstrap.cc:654] T 221cea23e1984729b30fa1a70ea2f23b P 616445a698fd48509f0ab763205ff942: Neither blocks nor log segments found. Creating new log.
I20260808 04:36:39.744516 355 log.cc:824] T 221cea23e1984729b30fa1a70ea2f23b P 616445a698fd48509f0ab763205ff942: Log is configured to *not* fsync() on all Append() calls
I20260808 04:36:39.749094 355 tablet_bootstrap.cc:492] T 221cea23e1984729b30fa1a70ea2f23b P 616445a698fd48509f0ab763205ff942: No bootstrap required, opened a new log
I20260808 04:36:39.749876 355 ts_tablet_manager.cc:1397] T 221cea23e1984729b30fa1a70ea2f23b P 616445a698fd48509f0ab763205ff942: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260808 04:36:39.755467 360 tablet_bootstrap.cc:492] T 221cea23e1984729b30fa1a70ea2f23b P bb13c0eed3064189b5ec30b86987761f: Bootstrap starting.
I20260808 04:36:39.755528 358 tablet_bootstrap.cc:492] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42: Bootstrap starting.
I20260808 04:36:39.758591 360 tablet_bootstrap.cc:654] T 221cea23e1984729b30fa1a70ea2f23b P bb13c0eed3064189b5ec30b86987761f: Neither blocks nor log segments found. Creating new log.
I20260808 04:36:39.758886 358 tablet_bootstrap.cc:654] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42: Neither blocks nor log segments found. Creating new log.
I20260808 04:36:39.759721 360 log.cc:824] T 221cea23e1984729b30fa1a70ea2f23b P bb13c0eed3064189b5ec30b86987761f: Log is configured to *not* fsync() on all Append() calls
I20260808 04:36:39.759889 358 log.cc:824] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42: Log is configured to *not* fsync() on all Append() calls
I20260808 04:36:39.761777 355 raft_consensus.cc:348] T 221cea23e1984729b30fa1a70ea2f23b P 616445a698fd48509f0ab763205ff942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616445a698fd48509f0ab763205ff942" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 32947 } } peers { permanent_uuid: "73f5eaca8b374bba8b75f00e8c954e42" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42425 } } peers { permanent_uuid: "bb13c0eed3064189b5ec30b86987761f" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34527 } }
I20260808 04:36:39.762173 355 raft_consensus.cc:374] T 221cea23e1984729b30fa1a70ea2f23b P 616445a698fd48509f0ab763205ff942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:36:39.762243 355 raft_consensus.cc:729] T 221cea23e1984729b30fa1a70ea2f23b P 616445a698fd48509f0ab763205ff942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616445a698fd48509f0ab763205ff942, State: Initialized, Role: FOLLOWER
I20260808 04:36:39.762770 355 consensus_queue.cc:260] T 221cea23e1984729b30fa1a70ea2f23b P 616445a698fd48509f0ab763205ff942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616445a698fd48509f0ab763205ff942" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 32947 } } peers { permanent_uuid: "73f5eaca8b374bba8b75f00e8c954e42" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42425 } } peers { permanent_uuid: "bb13c0eed3064189b5ec30b86987761f" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34527 } }
I20260808 04:36:39.764178 355 ts_tablet_manager.cc:1428] T 221cea23e1984729b30fa1a70ea2f23b P 616445a698fd48509f0ab763205ff942: Time spent starting tablet: real 0.014s user 0.003s sys 0.004s
I20260808 04:36:39.764684 32606 heartbeater.cc:503] Master 127.15.228.126:43681 was elected leader, sending a full tablet report...
I20260808 04:36:39.777719 358 tablet_bootstrap.cc:492] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42: No bootstrap required, opened a new log
I20260808 04:36:39.778059 358 ts_tablet_manager.cc:1397] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20260808 04:36:39.781503 358 raft_consensus.cc:348] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616445a698fd48509f0ab763205ff942" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 32947 } } peers { permanent_uuid: "73f5eaca8b374bba8b75f00e8c954e42" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42425 } } peers { permanent_uuid: "bb13c0eed3064189b5ec30b86987761f" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34527 } }
I20260808 04:36:39.781802 358 raft_consensus.cc:374] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:36:39.781857 358 raft_consensus.cc:729] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73f5eaca8b374bba8b75f00e8c954e42, State: Initialized, Role: FOLLOWER
I20260808 04:36:39.782387 358 consensus_queue.cc:260] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616445a698fd48509f0ab763205ff942" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 32947 } } peers { permanent_uuid: "73f5eaca8b374bba8b75f00e8c954e42" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42425 } } peers { permanent_uuid: "bb13c0eed3064189b5ec30b86987761f" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34527 } }
I20260808 04:36:39.783434 358 ts_tablet_manager.cc:1428] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260808 04:36:39.783864 332 heartbeater.cc:503] Master 127.15.228.126:43681 was elected leader, sending a full tablet report...
I20260808 04:36:39.785566 360 tablet_bootstrap.cc:492] T 221cea23e1984729b30fa1a70ea2f23b P bb13c0eed3064189b5ec30b86987761f: No bootstrap required, opened a new log
I20260808 04:36:39.785816 360 ts_tablet_manager.cc:1397] T 221cea23e1984729b30fa1a70ea2f23b P bb13c0eed3064189b5ec30b86987761f: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.005s
I20260808 04:36:39.790182 360 raft_consensus.cc:348] T 221cea23e1984729b30fa1a70ea2f23b P bb13c0eed3064189b5ec30b86987761f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616445a698fd48509f0ab763205ff942" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 32947 } } peers { permanent_uuid: "73f5eaca8b374bba8b75f00e8c954e42" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42425 } } peers { permanent_uuid: "bb13c0eed3064189b5ec30b86987761f" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34527 } }
I20260808 04:36:39.790616 360 raft_consensus.cc:374] T 221cea23e1984729b30fa1a70ea2f23b P bb13c0eed3064189b5ec30b86987761f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:36:39.790722 360 raft_consensus.cc:729] T 221cea23e1984729b30fa1a70ea2f23b P bb13c0eed3064189b5ec30b86987761f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb13c0eed3064189b5ec30b86987761f, State: Initialized, Role: FOLLOWER
I20260808 04:36:39.791503 360 consensus_queue.cc:260] T 221cea23e1984729b30fa1a70ea2f23b P bb13c0eed3064189b5ec30b86987761f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616445a698fd48509f0ab763205ff942" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 32947 } } peers { permanent_uuid: "73f5eaca8b374bba8b75f00e8c954e42" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42425 } } peers { permanent_uuid: "bb13c0eed3064189b5ec30b86987761f" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34527 } }
I20260808 04:36:39.792850 360 ts_tablet_manager.cc:1428] T 221cea23e1984729b30fa1a70ea2f23b P bb13c0eed3064189b5ec30b86987761f: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20260808 04:36:39.793191 32416 heartbeater.cc:503] Master 127.15.228.126:43681 was elected leader, sending a full tablet report...
W20260808 04:36:39.877476 333 tablet.cc:2369] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:36:39.902969 364 raft_consensus.cc:482] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:36:39.903271 364 raft_consensus.cc:504] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616445a698fd48509f0ab763205ff942" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 32947 } } peers { permanent_uuid: "73f5eaca8b374bba8b75f00e8c954e42" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42425 } } peers { permanent_uuid: "bb13c0eed3064189b5ec30b86987761f" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34527 } }
I20260808 04:36:39.906168 364 leader_election.cc:288] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616445a698fd48509f0ab763205ff942 (127.15.228.66:32947), bb13c0eed3064189b5ec30b86987761f (127.15.228.65:34527)
I20260808 04:36:39.912580 334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:39.906640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:32947 (local address 127.15.228.67:60385)
0808 04:36:39.907527 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0808 04:36:39.907545 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:36:39.907652 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:36:39.908102 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:36:39.908106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:36:39.908128 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:36:39.908255 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:36:39.908262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.909622 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.909626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:36:39.910994 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:36:39.911007 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.912016 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.912019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:36:39.912119 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:36:39.912142 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:36:39.912208 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 04:36:39.912406 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":78}
I20260808 04:36:39.912954 352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:39.907201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:40687 (local address 127.15.228.65:34527)
0808 04:36:39.907366 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 04:36:39.907371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:36:39.908044 (+ 673us) server_negotiation.cc:407] Connection header received
0808 04:36:39.908248 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:36:39.908251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:36:39.908312 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:36:39.908367 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:36:39.908830 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.909556 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.911582 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.912187 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.912290 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:36:39.912348 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 04:36:39.912724 (+ 376us) server_negotiation.cc:299] Negotiation successful
0808 04:36:39.912819 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:36:39.913375 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:39.907031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:34527 (local address 127.15.228.67:40687)
0808 04:36:39.907945 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0808 04:36:39.907961 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:36:39.908118 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:36:39.908525 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:36:39.908528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:36:39.908547 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:36:39.908671 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:36:39.908678 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.909715 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.909719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:36:39.911422 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:36:39.911433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.912474 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.912477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:36:39.912575 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:36:39.912598 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:36:39.913119 (+ 521us) client_negotiation.cc:240] Negotiation successful
0808 04:36:39.913202 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":2,"thread_start_us":150,"threads_started":1}
I20260808 04:36:39.913595 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:39.907551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:60385 (local address 127.15.228.66:32947)
0808 04:36:39.907688 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 04:36:39.907692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:36:39.907709 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:36:39.907774 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:36:39.907778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:36:39.907830 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:36:39.907946 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:36:39.908393 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.909439 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.911171 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.911777 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.911896 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:36:39.911954 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 04:36:39.913360 (+ 1406us) server_negotiation.cc:299] Negotiation successful
0808 04:36:39.913460 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 04:36:39.914292 32541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221cea23e1984729b30fa1a70ea2f23b" candidate_uuid: "73f5eaca8b374bba8b75f00e8c954e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616445a698fd48509f0ab763205ff942" is_pre_election: true
I20260808 04:36:39.914673 32541 raft_consensus.cc:2454] T 221cea23e1984729b30fa1a70ea2f23b P 616445a698fd48509f0ab763205ff942 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73f5eaca8b374bba8b75f00e8c954e42 in term 0.
I20260808 04:36:39.916685 32340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221cea23e1984729b30fa1a70ea2f23b" candidate_uuid: "73f5eaca8b374bba8b75f00e8c954e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb13c0eed3064189b5ec30b86987761f" is_pre_election: true
I20260808 04:36:39.917043 32340 raft_consensus.cc:2454] T 221cea23e1984729b30fa1a70ea2f23b P bb13c0eed3064189b5ec30b86987761f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73f5eaca8b374bba8b75f00e8c954e42 in term 0.
I20260808 04:36:39.918161 32641 leader_election.cc:302] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616445a698fd48509f0ab763205ff942, 73f5eaca8b374bba8b75f00e8c954e42; no voters:
I20260808 04:36:39.918541 364 raft_consensus.cc:2781] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:36:39.918622 364 raft_consensus.cc:482] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:36:39.918663 364 raft_consensus.cc:3037] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:36:39.920665 364 raft_consensus.cc:504] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616445a698fd48509f0ab763205ff942" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 32947 } } peers { permanent_uuid: "73f5eaca8b374bba8b75f00e8c954e42" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42425 } } peers { permanent_uuid: "bb13c0eed3064189b5ec30b86987761f" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34527 } }
I20260808 04:36:39.921103 364 leader_election.cc:288] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [CANDIDATE]: Term 1 election: Requested vote from peers 616445a698fd48509f0ab763205ff942 (127.15.228.66:32947), bb13c0eed3064189b5ec30b86987761f (127.15.228.65:34527)
I20260808 04:36:39.921933 32340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221cea23e1984729b30fa1a70ea2f23b" candidate_uuid: "73f5eaca8b374bba8b75f00e8c954e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb13c0eed3064189b5ec30b86987761f"
I20260808 04:36:39.922022 32541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221cea23e1984729b30fa1a70ea2f23b" candidate_uuid: "73f5eaca8b374bba8b75f00e8c954e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616445a698fd48509f0ab763205ff942"
I20260808 04:36:39.922097 32340 raft_consensus.cc:3037] T 221cea23e1984729b30fa1a70ea2f23b P bb13c0eed3064189b5ec30b86987761f [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:36:39.922161 32541 raft_consensus.cc:3037] T 221cea23e1984729b30fa1a70ea2f23b P 616445a698fd48509f0ab763205ff942 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:36:39.923569 32541 raft_consensus.cc:2454] T 221cea23e1984729b30fa1a70ea2f23b P 616445a698fd48509f0ab763205ff942 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73f5eaca8b374bba8b75f00e8c954e42 in term 1.
I20260808 04:36:39.923727 32340 raft_consensus.cc:2454] T 221cea23e1984729b30fa1a70ea2f23b P bb13c0eed3064189b5ec30b86987761f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73f5eaca8b374bba8b75f00e8c954e42 in term 1.
I20260808 04:36:39.923938 32641 leader_election.cc:302] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616445a698fd48509f0ab763205ff942, 73f5eaca8b374bba8b75f00e8c954e42; no voters:
I20260808 04:36:39.924495 364 raft_consensus.cc:2781] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:36:39.924734 364 raft_consensus.cc:686] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [term 1 LEADER]: Becoming Leader. State: Replica: 73f5eaca8b374bba8b75f00e8c954e42, State: Running, Role: LEADER
I20260808 04:36:39.925129 364 consensus_queue.cc:237] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616445a698fd48509f0ab763205ff942" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 32947 } } peers { permanent_uuid: "73f5eaca8b374bba8b75f00e8c954e42" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42425 } } peers { permanent_uuid: "bb13c0eed3064189b5ec30b86987761f" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34527 } }
I20260808 04:36:39.932402 32116 catalog_manager.cc:5495] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73f5eaca8b374bba8b75f00e8c954e42 (127.15.228.67). New cstate: current_term: 1 leader_uuid: "73f5eaca8b374bba8b75f00e8c954e42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616445a698fd48509f0ab763205ff942" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73f5eaca8b374bba8b75f00e8c954e42" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb13c0eed3064189b5ec30b86987761f" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34527 } health_report { overall_health: UNKNOWN } } }
W20260808 04:36:39.956375 32610 tablet.cc:2369] T 221cea23e1984729b30fa1a70ea2f23b P 616445a698fd48509f0ab763205ff942: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 04:36:39 dist-test-slave-drl0 krb5kdc[31448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163798, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
I20260808 04:36:39.976567 350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:39.970154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56932 (local address 127.15.228.67:42425)
0808 04:36:39.970342 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 04:36:39.970347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:36:39.970365 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:36:39.970469 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:36:39.970473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:36:39.970550 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:36:39.970674 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:36:39.971138 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.971939 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.973718 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:36:39.974079 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:36:39.974207 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:36:39.974778 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:36:39.975616 (+ 838us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:36:39.975766 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 04:36:39.976237 (+ 471us) server_negotiation.cc:299] Negotiation successful
0808 04:36:39.976337 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 04:36:39.987092 32340 raft_consensus.cc:1261] T 221cea23e1984729b30fa1a70ea2f23b P bb13c0eed3064189b5ec30b86987761f [term 1 FOLLOWER]: Refusing update from remote peer 73f5eaca8b374bba8b75f00e8c954e42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:36:39.992002 367 consensus_queue.cc:1035] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb13c0eed3064189b5ec30b86987761f" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:36:40.003386 32541 raft_consensus.cc:1261] T 221cea23e1984729b30fa1a70ea2f23b P 616445a698fd48509f0ab763205ff942 [term 1 FOLLOWER]: Refusing update from remote peer 73f5eaca8b374bba8b75f00e8c954e42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:36:40.004376 367 consensus_queue.cc:1035] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616445a698fd48509f0ab763205ff942" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:36:40.023903 375 mvcc.cc:205] Tried to move back new op lower bound from 7316126924734779392 to 7316126924496412672. Current Snapshot: MvccSnapshot[applied={T|T < 7316126924734779392}]
I20260808 04:36:40.025082 376 mvcc.cc:205] Tried to move back new op lower bound from 7316126924734779392 to 7316126924496412672. Current Snapshot: MvccSnapshot[applied={T|T < 7316126924734779392}]
I20260808 04:36:40.037670 378 mvcc.cc:205] Tried to move back new op lower bound from 7316126924734779392 to 7316126924496412672. Current Snapshot: MvccSnapshot[applied={T|T < 7316126924734779392}]
I20260808 04:36:40.043457 32116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:36:40.043740 32116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:36:40.047130 32116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fe36bd6e08c4a2289750d21a01b0630: Sending DeleteTablet for 3 replicas of tablet 221cea23e1984729b30fa1a70ea2f23b
I20260808 04:36:40.048516 32513 tablet_service.cc:1514] Processing DeleteTablet for tablet 221cea23e1984729b30fa1a70ea2f23b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:36:40 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:35846
I20260808 04:36:40.049077 32698 tablet_service.cc:1514] Processing DeleteTablet for tablet 221cea23e1984729b30fa1a70ea2f23b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:36:40 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:56920
I20260808 04:36:40.049646 32312 tablet_service.cc:1514] Processing DeleteTablet for tablet 221cea23e1984729b30fa1a70ea2f23b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:36:40 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:36348
I20260808 04:36:40.050200 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 32227
I20260808 04:36:40.075330 386 tablet_replica.cc:330] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42: stopping tablet replica
I20260808 04:36:40.076059 386 raft_consensus.cc:2229] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:36:40.077239 386 raft_consensus.cc:2258] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:36:40.079882 386 ts_tablet_manager.cc:1905] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:36:40.084781 386 ts_tablet_manager.cc:1918] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:36:40.085042 386 log.cc:1196] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/221cea23e1984729b30fa1a70ea2f23b
I20260808 04:36:40.085690 386 ts_tablet_manager.cc:1939] T 221cea23e1984729b30fa1a70ea2f23b P 73f5eaca8b374bba8b75f00e8c954e42: Deleting consensus metadata
I20260808 04:36:40.087432 32096 catalog_manager.cc:4841] TS 73f5eaca8b374bba8b75f00e8c954e42 (127.15.228.67:42425): tablet 221cea23e1984729b30fa1a70ea2f23b (table test-table [id=8e33d4b8a85d40be86f520ce42663a68]) successfully deleted
W20260808 04:36:40.099987 32098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.228.65:34527 (error 108)
I20260808 04:36:40.101423 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 32425
I20260808 04:36:40.114892 349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:40.114486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:34527 (local address 127.0.0.1:36358)
0808 04:36:40.114642 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 04:36:40.114758 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.65:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 04:36:40.115532 32098 catalog_manager.cc:4584] TS bb13c0eed3064189b5ec30b86987761f (127.15.228.65:34527): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 221cea23e1984729b30fa1a70ea2f23b: Network error: Client connection negotiation failed: client connection to 127.15.228.65:34527: connect: Connection refused (error 111)
I20260808 04:36:40.169287 349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:40.168872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:34527 (local address 127.0.0.1:36372)
0808 04:36:40.169102 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0808 04:36:40.169172 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.65:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260808 04:36:40.170233 349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:36:40.169986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:34527 (local address 127.0.0.1:36376)
0808 04:36:40.170119 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 04:36:40.170166 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.65:34527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260808 04:36:40.200892 349 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:36:40.200211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:32947 (local address 127.0.0.1:35852)
0808 04:36:40.200429 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 04:36:40.200445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:36:40.200544 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:36:40.200749 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.66:32947: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":134}
I20260808 04:36:40.204121 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 32616
I20260808 04:36:40.224133 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 32075
I20260808 04:36:40.251299 16273 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 04:36:40.610036 16273 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 04:36:40.610183 16273 mini_ranger.cc:66] Stopping Ranger...
I20260808 04:36:40.977362 16273 mini_ranger.cc:68] Stopped Ranger
2026-08-08 04:36:40.977 UTC [31492] LOG: received smart shutdown request
2026-08-08 04:36:40.982 UTC [31492] LOG: background worker "logical replication launcher" (PID 31502) exited with exit code 1
2026-08-08 04:36:40.983 UTC [31497] LOG: shutting down
2026-08-08 04:36:41.199 UTC [31492] LOG: database system is shut down
2026-08-08T04:36:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (58337 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:36:41 dist-test-slave-drl0 krb5kdc[406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:36:41 dist-test-slave-drl0 krb5kdc[406](info): set up 2 sockets
Aug 08 04:36:41 dist-test-slave-drl0 krb5kdc[406](info): commencing operation
krb5kdc: starting...
W20260808 04:36:42.641845 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.279s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:36:42 dist-test-slave-drl0 krb5kdc[406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:36:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:36:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.228.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.228.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.228.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/rangeradmin_127.15.228.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.228.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/rangeradmin_127.15.228.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.228.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.228.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.228.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/rangerlookup_127.15.228.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.228.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/rangerlookup_127.15.228.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.228.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.228.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.228.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/HTTP_127.15.228.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.228.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/HTTP_127.15.228.84@KRBTEST.COM.keytab.
I20260808 04:36:42.879408 16273 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhFaBKn/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/postgres -l logfile start
2026-08-08 04:36:45.962 UTC [445] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 04:36:45.962 UTC [445] LOG: listening on IPv4 address "127.15.228.84", port 45051
2026-08-08 04:36:45.968 UTC [445] LOG: listening on Unix socket "/tmp/.s.PGSQL.45051"
2026-08-08 04:36:45.996 UTC [449] LOG: database system was shut down at 2026-08-08 04:36:44 UTC
2026-08-08 04:36:46.001 UTC [445] LOG: database system is ready to accept connections
I20260808 04:36:47.209156 16273 mini_postgres.cc:96] Postgres bound to 45051
2026-08-08 04:36:47.218 UTC [464] FATAL: database "slave" does not exist
127.15.228.84:45051 - accepting connections
I20260808 04:36:47.219055 16273 mini_ranger.cc:161] Starting Ranger...
I20260808 04:36:47.238525 16273 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 04:36:47.860209 16273 mini_ranger.cc:87] Created ranger Postgres database
I20260808 04:36:47.860329 16273 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ranger-admin
2026-08-08 04:36:48,185 [I] DB FLAVOR :POSTGRES
2026-08-08 04:36:48,185 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 04:36:48,186 [I] Checking connection..
2026-08-08 04:36:48,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 04:36:48,484 [I] Checking connection passed.
2026-08-08 04:36:48,485 [I] --------- Verifying version history table ---------
2026-08-08 04:36:48,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:36:48,771 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 04:36:48,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:36:49,069 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 04:36:49,070 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 04:36:49,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 04:36:49.360 UTC [552] WARNING: there is no transaction in progress
2026-08-08 04:36:49,368 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 04:36:49,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 04:36:49,673 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 04:36:49,673 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 04:36:49,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 04:36:49,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 04:36:50,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-drl0.c.gcp-upstream.internal','N') ;"
2026-08-08 04:36:50,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 04:36:50,884 [I] Table x_portal_user does not exist in database ranger
2026-08-08 04:36:50,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 04:36:51,201 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 04:36:51,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 04:36:51,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 04:36:51,821 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 04:36:51,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 04:36:53.029 UTC [699] WARNING: there is no transaction in progress
2026-08-08 04:36:53.051 UTC [699] WARNING: there is no transaction in progress
2026-08-08 04:36:53.073 UTC [699] WARNING: there is no transaction in progress
2026-08-08 04:36:53.093 UTC [699] WARNING: there is no transaction in progress
2026-08-08 04:36:53.117 UTC [699] WARNING: there is no transaction in progress
2026-08-08 04:36:53.144 UTC [699] WARNING: there is no transaction in progress
2026-08-08 04:36:53.224 UTC [699] WARNING: there is no transaction in progress
2026-08-08 04:36:53.233 UTC [699] WARNING: there is no transaction in progress
2026-08-08 04:36:53.252 UTC [699] WARNING: there is no transaction in progress
2026-08-08 04:36:53.265 UTC [699] WARNING: there is no transaction in progress
2026-08-08 04:36:53.279 UTC [699] WARNING: there is no transaction in progress
2026-08-08 04:36:53.296 UTC [699] WARNING: there is no transaction in progress
2026-08-08 04:36:53.309 UTC [699] WARNING: there is no transaction in progress
2026-08-08 04:36:53.318 UTC [699] WARNING: there is no transaction in progress
2026-08-08 04:36:53.327 UTC [699] WARNING: there is no transaction in progress
2026-08-08 04:36:53.335 UTC [699] WARNING: there is no transaction in progress
2026-08-08 04:36:54,194 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 04:36:54,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 04:36:54,466 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 04:36:54,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 04:36:54,763 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 04:36:54,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 04:36:55,073 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 04:36:55,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 04:36:55,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 04:36:55,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-drl0.c.gcp-upstream.internal';"
2026-08-08 04:36:55,900 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 04:36:55,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 04:36:56,186 [I] DB_PATCHES have already been applied
I20260808 04:36:56.194532 16273 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskhFaBKn/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 04:36:56.194633 16273 mini_ranger.cc:193] Using host: 127.15.228.84
I20260808 04:36:56.200208 16273 mini_ranger.cc:227] Ranger admin URL: http://127.15.228.84:39815
Aug 08, 2026 4:36:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 4:36:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 4:36:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 4:36:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 4:36:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 4:36:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 4:36:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 04:36:57.578507 16273 mini_ranger.cc:160] Time spent starting Ranger: real 10.359s user 0.000s sys 0.015s
Aug 08, 2026 4:37:21 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 4:37:21 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 4:37:21 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 4:37:22 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 4:37:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 4:37:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 4:37:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 4:37:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 4:37:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 4:37:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 4:37:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 4:37:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 4:37:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 4:37:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 4:37:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 4:37:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 4:37:22 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 4:37:23 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 04:37:23.820119 16273 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.228.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.228.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.228.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/rangerkms_127.15.228.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.228.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/rangerkms_127.15.228.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.228.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.228.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.228.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/HTTP_127.15.228.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.228.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/HTTP_127.15.228.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 04:37:23 dist-test-slave-drl0 krb5kdc[406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163843, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 04:37:23.936746 16273 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 04:37:23.953825 16273 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 04:37:25.326871 16273 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 04:37:25.326993 16273 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ranger-kms
I20260808 04:37:25.327021 16273 mini_ranger_kms.cc:227] Using postgres at 127.15.228.84:45051
2026-08-08 04:37:25,601 [I] DB FLAVOR :POSTGRES
2026-08-08 04:37:25,601 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 04:37:25,601 [I] Checking connection
2026-08-08 04:37:25,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 04:37:25,919 [I] connection success
2026-08-08 04:37:25,919 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 04:37:25,919 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 04:37:25,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 04:37:26,226 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 04:37:26,227 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 04:37:26,227 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 04:37:26,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.228.84:45051/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 04:37:26,555 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 04:37:27.151702 16273 mini_ranger_kms.cc:310] Created kms service
I20260808 04:37:27.387768 16273 mini_ranger_kms.cc:326] Created kudu user
I20260808 04:37:27.446033 16273 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 04:37:27 dist-test-slave-drl0 krb5kdc[406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163843, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
I20260808 04:37:27.958513 16273 mini_ranger_kms.cc:384] Added ranger policy
I20260808 04:37:27.958684 16273 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhFaBKn/build/debug/bin/postgresql.jar:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhFaBKn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 04:37:27.958725 16273 mini_ranger_kms.cc:240] Using host: 127.15.228.84
I20260808 04:37:27.964993 16273 mini_ranger_kms.cc:276] Ranger KMS PID: 950
I20260808 04:37:27.965121 16273 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.228.84:42901
Aug 08, 2026 4:37:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 4:37:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 4:37:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 4:37:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 4:37:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 4:37:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 04:37:30 dist-test-slave-drl0 krb5kdc[406](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786163850, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.228.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:30.822829 16273 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.886s user 0.034s sys 0.216s
I20260808 04:37:30.823045 16273 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 04:37:30.823141 16273 mini_ranger_kms.cc:401] 127.15.228.84:42901/kms/v1/keys
Aug 08 04:37:31 dist-test-slave-drl0 krb5kdc[406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163850, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.228.84@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
Aug 08 04:37:31 dist-test-slave-drl0 krb5kdc[406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163850, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.228.84@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
Aug 08 04:37:34 dist-test-slave-drl0 krb5kdc[406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:37:34.275349 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:34959
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:34959
--ranger_config_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.228.84:42901/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:37:34.415251 1028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:34.415735 1028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:34.415853 1028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:34.420372 1028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:37:34.420449 1028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:34.420469 1028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:34.420486 1028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:37:34.420502 1028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:37:34.427599 1028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.228.84:42901/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:34959
--ranger_config_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:34959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:34.428915 1028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:34.430063 1028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:34.441205 1035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:34.441192 1034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:34.441181 1028 server_base.cc:1034] running on GCE node
W20260808 04:37:34.441167 1038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:34.442171 1028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:34.443476 1028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:34.444716 1028 hybrid_clock.cc:648] HybridClock initialized: now 1786163854444658 us; error 71 us; skew 500 ppm
Aug 08 04:37:34 dist-test-slave-drl0 krb5kdc[406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163854, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:34.448798 1028 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:37:34.449925 1028 webserver.cc:460] Webserver started at http://127.15.228.126:45575/ using document root <none> and password file <none>
I20260808 04:37:34.450627 1028 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:34.450726 1028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:37:34.451058 1028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:37:34 dist-test-slave-drl0 krb5kdc[406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163854, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
I20260808 04:37:34.522644 1028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "aad7a100f6844835bc4641e9ef8c9a63"
format_stamp: "Formatted at 2026-08-08 04:37:34 on dist-test-slave-drl0"
server_key: "cbb306227ec60e816f1c332be3f0caf9"
server_key_iv: "3a402fd25ea1295bc881e503453a87c1"
server_key_version: "kuduclusterkey@0"
I20260808 04:37:34.523622 1028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "aad7a100f6844835bc4641e9ef8c9a63"
format_stamp: "Formatted at 2026-08-08 04:37:34 on dist-test-slave-drl0"
server_key: "cbb306227ec60e816f1c332be3f0caf9"
server_key_iv: "3a402fd25ea1295bc881e503453a87c1"
server_key_version: "kuduclusterkey@0"
I20260808 04:37:34.527894 1028 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:37:34.551156 1049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:34.553140 1028 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260808 04:37:34.553442 1028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "aad7a100f6844835bc4641e9ef8c9a63"
format_stamp: "Formatted at 2026-08-08 04:37:34 on dist-test-slave-drl0"
server_key: "cbb306227ec60e816f1c332be3f0caf9"
server_key_iv: "3a402fd25ea1295bc881e503453a87c1"
server_key_version: "kuduclusterkey@0"
I20260808 04:37:34.553701 1028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:34.582814 1028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:34.586436 1028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:34.586679 1028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:34.597412 1028 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:34959
I20260808 04:37:34.597678 1117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:34959 every 8 connection(s)
I20260808 04:37:34.598905 1028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:37:34.603305 1118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:37:34.607892 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 1028
I20260808 04:37:34.608027 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:37:34.608448 16273 external_mini_cluster.cc:1428] Setting key e1992c0854ec24ab45361901c9dae0d3
I20260808 04:37:34.611939 1118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63: Bootstrap starting.
I20260808 04:37:34.617206 1118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63: Neither blocks nor log segments found. Creating new log.
I20260808 04:37:34.618296 1118 log.cc:824] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:37:34 dist-test-slave-drl0 krb5kdc[406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:37:34.622319 1118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63: No bootstrap required, opened a new log
I20260808 04:37:34.626194 1118 raft_consensus.cc:348] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad7a100f6844835bc4641e9ef8c9a63" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34959 } }
I20260808 04:37:34.626511 1118 raft_consensus.cc:374] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:37:34.626645 1118 raft_consensus.cc:729] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad7a100f6844835bc4641e9ef8c9a63, State: Initialized, Role: FOLLOWER
I20260808 04:37:34.627262 1118 consensus_queue.cc:260] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad7a100f6844835bc4641e9ef8c9a63" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34959 } }
I20260808 04:37:34.627498 1118 raft_consensus.cc:388] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:37:34.627580 1118 raft_consensus.cc:482] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:37:34.627691 1118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:37:34.629024 1118 raft_consensus.cc:504] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad7a100f6844835bc4641e9ef8c9a63" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34959 } }
I20260808 04:37:34.629488 1118 leader_election.cc:302] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aad7a100f6844835bc4641e9ef8c9a63; no voters:
I20260808 04:37:34.629877 1121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:34.610141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37470 (local address 127.15.228.126:34959)
0808 04:37:34.610620 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:34.610638 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:34.610675 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:37:34.611459 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:34.611479 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:34.611835 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:34.612186 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:34.613134 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:34.615385 (+ 2251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:34.617536 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:34.618303 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:34.618542 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:34.622739 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:34.622747 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:34.622770 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:34.622817 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:34.625665 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:34.626347 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:34.626358 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:34.626363 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:34.626492 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:34.626917 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:34.626923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:34.626924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:34.627627 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:34.627855 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:34.628426 (+ 571us) server_negotiation.cc:299] Negotiation successful
0808 04:37:34.628754 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":160,"threads_started":1}
I20260808 04:37:34.629882 1118 leader_election.cc:288] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:37:34.630194 1123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:37:34.630440 1123 raft_consensus.cc:686] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [term 1 LEADER]: Becoming Leader. State: Replica: aad7a100f6844835bc4641e9ef8c9a63, State: Running, Role: LEADER
I20260808 04:37:34.630774 1123 consensus_queue.cc:237] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad7a100f6844835bc4641e9ef8c9a63" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34959 } }
I20260808 04:37:34.631357 1118 sys_catalog.cc:564] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:37:34.632115 1123 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aad7a100f6844835bc4641e9ef8c9a63. Latest consensus state: current_term: 1 leader_uuid: "aad7a100f6844835bc4641e9ef8c9a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad7a100f6844835bc4641e9ef8c9a63" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34959 } } }
I20260808 04:37:34.632323 1123 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [sys.catalog]: This master's current role is: LEADER
I20260808 04:37:34.633136 1141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:37:34.633093 1126 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aad7a100f6844835bc4641e9ef8c9a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad7a100f6844835bc4641e9ef8c9a63" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34959 } } }
I20260808 04:37:34.633245 1126 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [sys.catalog]: This master's current role is: LEADER
I20260808 04:37:34.637100 1141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:37:34.637684 1118 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 04:37:34.652778 1141 catalog_manager.cc:1347] Generated new cluster ID: beffe57dea5943f4a3499c18e1b5d26c
I20260808 04:37:34.652904 1141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:37:34.668673 1141 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:37:34.669741 1141 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:37:34.677826 1141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63: Generated new TSK 0
I20260808 04:37:34.679301 1141 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
I20260808 04:37:35.471834 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.228.84:42901/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:34959
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:37:35.646684 1185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:35.647084 1185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:35.647186 1185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:35.654448 1185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:35.654546 1185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:35.654644 1185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:37:35.664860 1185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.228.84:42901/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:34959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:35.667161 1185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:35.669021 1185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:35.683712 1198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:35.684226 1203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:35.684242 1199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:35.684765 1185 server_base.cc:1034] running on GCE node
I20260808 04:37:35.685328 1185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:35.686287 1185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:35.687530 1185 hybrid_clock.cc:648] HybridClock initialized: now 1786163855687501 us; error 44 us; skew 500 ppm
Aug 08 04:37:35 dist-test-slave-drl0 krb5kdc[406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163855, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:35.691773 1185 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:37:35.692992 1185 webserver.cc:460] Webserver started at http://127.15.228.65:34643/ using document root <none> and password file <none>
I20260808 04:37:35.693926 1185 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:35.694077 1185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:37:35.694386 1185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:37:35 dist-test-slave-drl0 krb5kdc[406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163855, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
I20260808 04:37:35.714600 1185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "ddf34562e21a49509ee8c375e7b357f4"
format_stamp: "Formatted at 2026-08-08 04:37:35 on dist-test-slave-drl0"
server_key: "8433912445fd36b4a92604df57e33948"
server_key_iv: "70a1e1fa88a08a2681f3de7ba593b386"
server_key_version: "kuduclusterkey@0"
I20260808 04:37:35.715636 1185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "ddf34562e21a49509ee8c375e7b357f4"
format_stamp: "Formatted at 2026-08-08 04:37:35 on dist-test-slave-drl0"
server_key: "8433912445fd36b4a92604df57e33948"
server_key_iv: "70a1e1fa88a08a2681f3de7ba593b386"
server_key_version: "kuduclusterkey@0"
I20260808 04:37:35.729038 1185 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.000s
I20260808 04:37:35.766774 1214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:35.781601 1185 fs_manager.cc:725] Time spent opening block manager: real 0.029s user 0.003s sys 0.000s
I20260808 04:37:35.781828 1185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "ddf34562e21a49509ee8c375e7b357f4"
format_stamp: "Formatted at 2026-08-08 04:37:35 on dist-test-slave-drl0"
server_key: "8433912445fd36b4a92604df57e33948"
server_key_iv: "70a1e1fa88a08a2681f3de7ba593b386"
server_key_version: "kuduclusterkey@0"
I20260808 04:37:35.782003 1185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:35.797664 1185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:35.802217 1185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:35.802595 1185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:35.803498 1185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:37:35.804656 1185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:37:35.804769 1185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:35.804920 1185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:37:35.805022 1185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:35.816924 1185 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:35951
I20260808 04:37:35.818310 1185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:37:35.821880 1369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:35951 every 8 connection(s)
I20260808 04:37:35.823750 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 1185
I20260808 04:37:35.823908 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:37:35.824246 16273 external_mini_cluster.cc:1428] Setting key ae19bb0e6fd71c9e830c2ef57dc91362
Aug 08 04:37:35 dist-test-slave-drl0 krb5kdc[406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163855, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:37:35.847816 1374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:35.826061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:54521 (local address 127.15.228.126:34959)
0808 04:37:35.829349 (+ 3288us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:35.829356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:35.829373 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:37:35.829418 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:35.829424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:35.829486 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:35.829582 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:35.831258 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:35.832363 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:35.834731 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:35.835152 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:35.835241 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:35.842700 (+ 7459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:35.842705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:35.842709 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:35.842747 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:35.844816 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:35.845586 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:35.845592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:35.845597 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:35.845703 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:35.846177 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:35.846181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:35.846183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:35.846458 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:35.846553 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:35.847419 (+ 866us) server_negotiation.cc:299] Negotiation successful
0808 04:37:35.847590 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3182,"thread_start_us":122,"threads_started":1}
I20260808 04:37:35.850003 1372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:35.824650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34959 (local address 127.15.228.65:54521)
0808 04:37:35.827530 (+ 2880us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:35.827593 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:35.828878 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:35.829972 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:35.829982 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:35.830348 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:35.831085 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:35.831105 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:35.832697 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:35.832702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:35.834515 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:35.834538 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:35.835720 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:35.835724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:35.835968 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:35.836731 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:35.836755 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:35.839508 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:35.844996 (+ 5488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:35.845003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:35.845024 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:35.845449 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:35.845828 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:35.845831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:35.845833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:35.846049 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:35.846627 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:35.846635 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:35.847081 (+ 446us) client_negotiation.cc:769] Sending connection context
0808 04:37:35.848408 (+ 1327us) client_negotiation.cc:240] Negotiation successful
0808 04:37:35.848824 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2637,"thread_start_us":178,"threads_started":1}
Aug 08 04:37:35 dist-test-slave-drl0 krb5kdc[406](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786163855, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:35.853175 1370 heartbeater.cc:347] Connected to a master server at 127.15.228.126:34959
I20260808 04:37:35.853456 1370 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:35.854130 1370 heartbeater.cc:511] Master 127.15.228.126:34959 requested a full tablet report, sending...
I20260808 04:37:35.856886 1079 ts_manager.cc:194] Registered new tserver with Master: ddf34562e21a49509ee8c375e7b357f4 (127.15.228.65:35951)
I20260808 04:37:35.859225 1079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:54521
WARNING: no policy specified for kudu/127.15.228.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
I20260808 04:37:35.921654 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.228.84:42901/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:34959
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:37:36.095067 1379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:36.095556 1379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:36.095685 1379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:36.100898 1379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:36.101009 1379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:36.101117 1379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:37:36.108078 1379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.228.84:42901/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:34959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:36.109779 1379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:36.111485 1379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:36.129057 1389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:36.129673 1392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:36.129890 1379 server_base.cc:1034] running on GCE node
W20260808 04:37:36.129997 1390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:36.130610 1379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:36.131479 1379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:36.132736 1379 hybrid_clock.cc:648] HybridClock initialized: now 1786163856132666 us; error 82 us; skew 500 ppm
Aug 08 04:37:36 dist-test-slave-drl0 krb5kdc[406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163856, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:36.139811 1379 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:37:36.141278 1379 webserver.cc:460] Webserver started at http://127.15.228.66:46793/ using document root <none> and password file <none>
I20260808 04:37:36.142064 1379 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:36.142179 1379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:37:36.142447 1379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:37:36 dist-test-slave-drl0 krb5kdc[406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163856, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
I20260808 04:37:36.176985 1379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "cf09a18f254848e9807ce04b266532ae"
format_stamp: "Formatted at 2026-08-08 04:37:36 on dist-test-slave-drl0"
server_key: "cc3efe768c65ce2847e6e58ca328ed9b"
server_key_iv: "b22bb7b85ba7776335fc0b12f876576d"
server_key_version: "kuduclusterkey@0"
I20260808 04:37:36.178150 1379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "cf09a18f254848e9807ce04b266532ae"
format_stamp: "Formatted at 2026-08-08 04:37:36 on dist-test-slave-drl0"
server_key: "cc3efe768c65ce2847e6e58ca328ed9b"
server_key_iv: "b22bb7b85ba7776335fc0b12f876576d"
server_key_version: "kuduclusterkey@0"
I20260808 04:37:36.186573 1379 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260808 04:37:36.207619 1401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:36.209470 1379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:37:36.209623 1379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "cf09a18f254848e9807ce04b266532ae"
format_stamp: "Formatted at 2026-08-08 04:37:36 on dist-test-slave-drl0"
server_key: "cc3efe768c65ce2847e6e58ca328ed9b"
server_key_iv: "b22bb7b85ba7776335fc0b12f876576d"
server_key_version: "kuduclusterkey@0"
I20260808 04:37:36.209790 1379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:36.238598 1379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:36.242619 1379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:36.242884 1379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:36.243729 1379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:37:36.244899 1379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:37:36.245023 1379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:36.245124 1379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:37:36.245188 1379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:36.258026 1379 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:34713
I20260808 04:37:36.259536 1379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:37:36.263468 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 1379
I20260808 04:37:36.263607 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:37:36.263955 16273 external_mini_cluster.cc:1428] Setting key e614d45ca64fe4026dcccfa68902c7b1
I20260808 04:37:36.271731 1561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:34713 every 8 connection(s)
WARNING: no policy specified for kudu/127.15.228.67@KRBTEST.COM; defaulting to no policy
Aug 08 04:37:36 dist-test-slave-drl0 krb5kdc[406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163856, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:37:36.305436 1374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:36.275442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:50451 (local address 127.15.228.126:34959)
0808 04:37:36.277298 (+ 1856us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:36.277303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:36.277807 (+ 504us) server_negotiation.cc:407] Connection header received
0808 04:37:36.279256 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:36.279260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:36.279328 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:36.279407 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:36.283891 (+ 4484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:36.284775 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:36.286683 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:36.287103 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:36.287190 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:36.300067 (+ 12877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:36.300073 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:36.300076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:36.300117 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:36.302198 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:36.303089 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:36.303093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:36.303095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:36.303195 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:36.303692 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:36.303695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:36.303697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:36.303979 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:36.304204 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:36.305050 (+ 846us) server_negotiation.cc:299] Negotiation successful
0808 04:37:36.305223 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1724}
I20260808 04:37:36.307667 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:36.276049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34959 (local address 127.15.228.66:50451)
0808 04:37:36.277628 (+ 1579us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:36.277690 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:36.278965 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:36.279868 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:36.279878 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:36.280300 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:36.281077 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:36.281099 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:36.284985 (+ 3886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:36.284989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:36.286515 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:36.286529 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:36.287440 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:36.287444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:36.288600 (+ 1156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:36.294274 (+ 5674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:36.294304 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:36.299744 (+ 5440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:36.302423 (+ 2679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:36.302432 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:36.302462 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:36.302919 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:36.303336 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:36.303339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:36.303341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:36.303566 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:36.304302 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:36.304309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:36.304703 (+ 394us) client_negotiation.cc:769] Sending connection context
0808 04:37:36.306002 (+ 1299us) client_negotiation.cc:240] Negotiation successful
0808 04:37:36.306418 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1331,"thread_start_us":204,"threads_started":1}
I20260808 04:37:36.309566 1563 heartbeater.cc:347] Connected to a master server at 127.15.228.126:34959
I20260808 04:37:36.309841 1563 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:36.310503 1563 heartbeater.cc:511] Master 127.15.228.126:34959 requested a full tablet report, sending...
I20260808 04:37:36.311841 1079 ts_manager.cc:194] Registered new tserver with Master: cf09a18f254848e9807ce04b266532ae (127.15.228.66:34713)
I20260808 04:37:36.312675 1079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:50451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
I20260808 04:37:36.340350 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.228.84:42901/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:34959
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:37:36 dist-test-slave-drl0 krb5kdc[406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163855, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
W20260808 04:37:36.529677 1570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:36.530159 1570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:36.530278 1570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:36.537545 1570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:36.537739 1570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:36.537890 1570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:37:36.546825 1570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.228.84:42901/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:34959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:36.548617 1570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:36.550137 1570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:36.564765 1579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:36.564765 1583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:36.565238 1577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:36.567117 1570 server_base.cc:1034] running on GCE node
I20260808 04:37:36.567715 1570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:36.568457 1570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:36.569679 1570 hybrid_clock.cc:648] HybridClock initialized: now 1786163856569638 us; error 50 us; skew 500 ppm
Aug 08 04:37:36 dist-test-slave-drl0 krb5kdc[406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163856, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:36.572817 1570 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:37:36.573994 1570 webserver.cc:460] Webserver started at http://127.15.228.67:37783/ using document root <none> and password file <none>
I20260808 04:37:36.574627 1570 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:36.574733 1570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:37:36.575078 1570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 04:37:36 dist-test-slave-drl0 krb5kdc[406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163856, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
I20260808 04:37:36.607322 1570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "b606b0ddb1604441b7b0d07b4dcaea61"
format_stamp: "Formatted at 2026-08-08 04:37:36 on dist-test-slave-drl0"
server_key: "0cb957a58a7b5c8d9c0411b7c85d87f0"
server_key_iv: "816a1903a2f93a0fa9c0c2b692d0e49c"
server_key_version: "kuduclusterkey@0"
I20260808 04:37:36.610476 1570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "b606b0ddb1604441b7b0d07b4dcaea61"
format_stamp: "Formatted at 2026-08-08 04:37:36 on dist-test-slave-drl0"
server_key: "0cb957a58a7b5c8d9c0411b7c85d87f0"
server_key_iv: "816a1903a2f93a0fa9c0c2b692d0e49c"
server_key_version: "kuduclusterkey@0"
I20260808 04:37:36.619029 1570 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.000s
I20260808 04:37:36.642879 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:36.644939 1570 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260808 04:37:36.645121 1570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "b606b0ddb1604441b7b0d07b4dcaea61"
format_stamp: "Formatted at 2026-08-08 04:37:36 on dist-test-slave-drl0"
server_key: "0cb957a58a7b5c8d9c0411b7c85d87f0"
server_key_iv: "816a1903a2f93a0fa9c0c2b692d0e49c"
server_key_version: "kuduclusterkey@0"
I20260808 04:37:36.645287 1570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:36.671267 1570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:36.675216 1570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:36.675449 1570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:36.676287 1570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:37:36.677485 1570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:37:36.677776 1570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:36.677884 1570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:37:36.677919 1570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:36.692940 1570 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:39573
I20260808 04:37:36.693305 1771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:39573 every 8 connection(s)
I20260808 04:37:36.695034 1570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:37:36.704658 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 1570
I20260808 04:37:36.704973 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:37:36.705356 16273 external_mini_cluster.cc:1428] Setting key 26937d8fa05176a7b62e3b9de277adda
Aug 08 04:37:36 dist-test-slave-drl0 krb5kdc[406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163856, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:37:36.715626 1374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:36.698653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:60087 (local address 127.15.228.126:34959)
0808 04:37:36.698820 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:36.698826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:36.700474 (+ 1648us) server_negotiation.cc:407] Connection header received
0808 04:37:36.701932 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:36.701937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:36.702024 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:36.702139 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:36.703853 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:36.704601 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:36.707180 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:36.707568 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:36.707637 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:36.711118 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:36.711122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:36.711124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:36.711151 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:36.713090 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:36.713711 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:36.713721 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:36.713722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:36.713805 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:36.714218 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:36.714222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:36.714224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:36.714454 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:36.714581 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:36.715266 (+ 685us) server_negotiation.cc:299] Negotiation successful
0808 04:37:36.715434 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:37:36.717939 1774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:36.699299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34959 (local address 127.15.228.67:60087)
0808 04:37:36.700183 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:36.700280 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:36.701608 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:36.702393 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:36.702403 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:36.702826 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:36.703635 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:36.703656 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:36.704772 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:36.704776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:36.706471 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:36.706496 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:36.708466 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:36.708472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:36.708695 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:36.709372 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:36.709391 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:36.710862 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:36.713232 (+ 2370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:36.713240 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:36.713260 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:36.713581 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:36.713916 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:36.713920 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:36.713922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:36.714098 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:36.714582 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:36.714591 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:36.714920 (+ 329us) client_negotiation.cc:769] Sending connection context
0808 04:37:36.715205 (+ 285us) client_negotiation.cc:240] Negotiation successful
0808 04:37:36.716284 (+ 1079us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":218,"threads_started":1}
I20260808 04:37:36.719332 1772 heartbeater.cc:347] Connected to a master server at 127.15.228.126:34959
I20260808 04:37:36.719559 1772 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:36.720225 1772 heartbeater.cc:511] Master 127.15.228.126:34959 requested a full tablet report, sending...
I20260808 04:37:36.721484 1079 ts_manager.cc:194] Registered new tserver with Master: b606b0ddb1604441b7b0d07b4dcaea61 (127.15.228.67:39573)
I20260808 04:37:36.722255 1079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:60087
I20260808 04:37:36.732618 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:37:36.733639 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 1185
I20260808 04:37:36.786729 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 1379
I20260808 04:37:36.798899 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 1570
I20260808 04:37:36.825672 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 1028
I20260808 04:37:36.840499 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:34959
--webserver_interface=127.15.228.126
--webserver_port=45575
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:34959
--ranger_config_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.228.84:42901/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.228.84:42901/kms with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:37:36.978282 1775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:36.978827 1775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:36.978870 1775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:36.984117 1775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:37:36.984193 1775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:36.984215 1775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:36.984231 1775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:37:36.984339 1775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:37:36.990743 1775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.228.84:42901/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:34959
--ranger_config_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:34959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=45575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:36.992313 1775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:36.993626 1775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:37.005677 1780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:37.005772 1781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:37.005741 1784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:37.006008 1775 server_base.cc:1034] running on GCE node
I20260808 04:37:37.007299 1775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:37.009017 1775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:37.010308 1775 hybrid_clock.cc:648] HybridClock initialized: now 1786163857010265 us; error 64 us; skew 500 ppm
Aug 08 04:37:37 dist-test-slave-drl0 krb5kdc[406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163857, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:37.013628 1775 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:37:37.014667 1775 webserver.cc:460] Webserver started at http://127.15.228.126:45575/ using document root <none> and password file <none>
I20260808 04:37:37.015415 1775 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:37.015529 1775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 04:37:37 dist-test-slave-drl0 krb5kdc[406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163857, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
I20260808 04:37:37.092133 1775 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:37:37.095110 1792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:37.097232 1775 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:37:37.097433 1775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "aad7a100f6844835bc4641e9ef8c9a63"
format_stamp: "Formatted at 2026-08-08 04:37:34 on dist-test-slave-drl0"
server_key: "cbb306227ec60e816f1c332be3f0caf9"
server_key_iv: "3a402fd25ea1295bc881e503453a87c1"
server_key_version: "kuduclusterkey@0"
I20260808 04:37:37.098209 1775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:37.115278 1775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:37.118800 1775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:37.119051 1775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:37.127190 1775 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:34959
I20260808 04:37:37.127218 1868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:34959 every 8 connection(s)
I20260808 04:37:37.128409 1775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:37:37.131162 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 1775
I20260808 04:37:37.131664 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.228.84:42901/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:35951
--local_ip_for_outbound_sockets=127.15.228.65
--tserver_master_addrs=127.15.228.126:34959
--webserver_port=34643
--webserver_interface=127.15.228.65
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.228.84:42901/kms with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:37:37.133471 1869 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:37:37.135335 1869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63: Bootstrap starting.
I20260808 04:37:37.156106 1869 log.cc:824] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63: Log is configured to *not* fsync() on all Append() calls
I20260808 04:37:37.164541 1869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:37:37.165037 1869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63: Bootstrap complete.
I20260808 04:37:37.169611 1869 raft_consensus.cc:348] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad7a100f6844835bc4641e9ef8c9a63" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34959 } }
I20260808 04:37:37.170063 1869 raft_consensus.cc:729] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad7a100f6844835bc4641e9ef8c9a63, State: Initialized, Role: FOLLOWER
I20260808 04:37:37.170626 1869 consensus_queue.cc:260] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad7a100f6844835bc4641e9ef8c9a63" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34959 } }
I20260808 04:37:37.170781 1869 raft_consensus.cc:388] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:37:37.170874 1869 raft_consensus.cc:482] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:37:37.171030 1869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:37:37.174082 1869 raft_consensus.cc:504] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad7a100f6844835bc4641e9ef8c9a63" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34959 } }
I20260808 04:37:37.174499 1869 leader_election.cc:302] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aad7a100f6844835bc4641e9ef8c9a63; no voters:
I20260808 04:37:37.174916 1869 leader_election.cc:288] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 04:37:37.175060 1873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:37:37.175295 1873 raft_consensus.cc:686] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [term 2 LEADER]: Becoming Leader. State: Replica: aad7a100f6844835bc4641e9ef8c9a63, State: Running, Role: LEADER
I20260808 04:37:37.175715 1873 consensus_queue.cc:237] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad7a100f6844835bc4641e9ef8c9a63" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34959 } }
I20260808 04:37:37.176371 1869 sys_catalog.cc:564] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:37:37.177305 1874 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "aad7a100f6844835bc4641e9ef8c9a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad7a100f6844835bc4641e9ef8c9a63" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34959 } } }
I20260808 04:37:37.177462 1874 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [sys.catalog]: This master's current role is: LEADER
I20260808 04:37:37.177897 1875 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aad7a100f6844835bc4641e9ef8c9a63. Latest consensus state: current_term: 2 leader_uuid: "aad7a100f6844835bc4641e9ef8c9a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad7a100f6844835bc4641e9ef8c9a63" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34959 } } }
I20260808 04:37:37.178057 1875 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63 [sys.catalog]: This master's current role is: LEADER
I20260808 04:37:37.179421 1884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:37:37.179908 1869 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 04:37:37.186714 1884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:37:37.187906 1884 catalog_manager.cc:1259] Loaded cluster ID: beffe57dea5943f4a3499c18e1b5d26c
I20260808 04:37:37.188021 1884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:37:37.191334 1884 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:37:37.192157 1884 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63: Loaded TSK: 0
I20260808 04:37:37.193801 1884 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:37:37.308066 1871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:37.308450 1871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:37.308523 1871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:37.314867 1871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:37.315033 1871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:37.315160 1871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:37:37.324400 1871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.228.84:42901/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:35951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=34643
--tserver_master_addrs=127.15.228.126:34959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:37.325927 1871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:37.327490 1871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:37.342533 1920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:37.343400 1919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:37.343917 1923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:37.345672 1871 server_base.cc:1034] running on GCE node
I20260808 04:37:37.346243 1871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:37.347146 1871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:37.348351 1871 hybrid_clock.cc:648] HybridClock initialized: now 1786163857348332 us; error 55 us; skew 500 ppm
Aug 08 04:37:37 dist-test-slave-drl0 krb5kdc[406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163857, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:37.353719 1871 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:37:37.355022 1871 webserver.cc:460] Webserver started at http://127.15.228.65:34643/ using document root <none> and password file <none>
I20260808 04:37:37.355659 1871 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:37.355777 1871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 04:37:37 dist-test-slave-drl0 krb5kdc[406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163857, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
I20260808 04:37:37.385468 1871 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.000s
I20260808 04:37:37.391918 1933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:37.393895 1871 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260808 04:37:37.394074 1871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "ddf34562e21a49509ee8c375e7b357f4"
format_stamp: "Formatted at 2026-08-08 04:37:35 on dist-test-slave-drl0"
server_key: "8433912445fd36b4a92604df57e33948"
server_key_iv: "70a1e1fa88a08a2681f3de7ba593b386"
server_key_version: "kuduclusterkey@0"
I20260808 04:37:37.394963 1871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:37.426381 1871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:37.430644 1871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:37.430982 1871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:37.431795 1871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:37:37.433843 1871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:37:37.434065 1871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 04:37:37.434239 1871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:37:37.434330 1871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:37.451429 1871 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:35951
I20260808 04:37:37.453087 1871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:37:37.457878 2093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:35951 every 8 connection(s)
I20260808 04:37:37.460386 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 1871
I20260808 04:37:37.460973 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.228.84:42901/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:34713
--local_ip_for_outbound_sockets=127.15.228.66
--tserver_master_addrs=127.15.228.126:34959
--webserver_port=46793
--webserver_interface=127.15.228.66
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.228.84:42901/kms with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:37:37 dist-test-slave-drl0 krb5kdc[406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163857, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:37:37.497896 2098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:37.462752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34959 (local address 127.15.228.65:33311)
0808 04:37:37.464269 (+ 1517us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:37.464336 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:37.465620 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:37.469996 (+ 4376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:37.470006 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:37.470393 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:37.471137 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:37.471161 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:37.478184 (+ 7023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:37.478191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:37.479787 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:37.479806 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:37.481315 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:37.481319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:37.481513 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:37.488428 (+ 6915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:37.488460 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:37.490340 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:37.493346 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:37.493354 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:37.493372 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:37.493981 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:37.494498 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:37.494501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:37.494504 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:37.494694 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:37.495717 (+ 1023us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:37.495725 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:37.496087 (+ 362us) client_negotiation.cc:769] Sending connection context
0808 04:37:37.496316 (+ 229us) client_negotiation.cc:240] Negotiation successful
0808 04:37:37.496718 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1230,"thread_start_us":199,"threads_started":1}
I20260808 04:37:37.499965 2101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:37.463693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:33311 (local address 127.15.228.126:34959)
0808 04:37:37.468177 (+ 4484us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:37.468201 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:37.468250 (+ 49us) server_negotiation.cc:407] Connection header received
0808 04:37:37.468582 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:37.468598 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:37.469052 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:37.469446 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:37.471366 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:37.472966 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:37.480009 (+ 7043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:37.480819 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:37.481039 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:37.490688 (+ 9649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:37.490697 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:37.490720 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:37.490764 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:37.493155 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:37.494156 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:37.494164 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:37.494170 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:37.494358 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:37.494816 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:37.494820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:37.494822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:37.495505 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:37.495652 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:37.498639 (+ 2987us) server_negotiation.cc:299] Negotiation successful
0808 04:37:37.499029 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4296,"thread_start_us":171,"threads_started":1}
I20260808 04:37:37.504969 2094 heartbeater.cc:347] Connected to a master server at 127.15.228.126:34959
I20260808 04:37:37.505267 2094 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:37.506006 2094 heartbeater.cc:511] Master 127.15.228.126:34959 requested a full tablet report, sending...
W20260808 04:37:37.508692 2094 heartbeater.cc:644] Failed to heartbeat to 127.15.228.126:34959 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 04:37:37.508832 2094 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:37.509054 2094 heartbeater.cc:511] Master 127.15.228.126:34959 requested a full tablet report, sending...
I20260808 04:37:37.510138 2094 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:37.510324 2094 heartbeater.cc:511] Master 127.15.228.126:34959 requested a full tablet report, sending...
W20260808 04:37:37.511248 2094 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:37:37.670953 2097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:37.671365 2097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:37.671439 2097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:37.677490 2097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:37.677637 2097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:37.677759 2097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:37:37.686269 2097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.228.84:42901/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:34713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=46793
--tserver_master_addrs=127.15.228.126:34959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:37.687834 2097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:37.689216 2097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:37.704244 2107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:37.704434 2106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:37.704461 2109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:37.712319 2097 server_base.cc:1034] running on GCE node
I20260808 04:37:37.713037 2097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:37.714015 2097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:37.721560 2097 hybrid_clock.cc:648] HybridClock initialized: now 1786163857721502 us; error 65 us; skew 500 ppm
Aug 08 04:37:37 dist-test-slave-drl0 krb5kdc[406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163857, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:37.727030 2097 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:37:37.728348 2097 webserver.cc:460] Webserver started at http://127.15.228.66:46793/ using document root <none> and password file <none>
I20260808 04:37:37.729168 2097 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:37.729296 2097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 04:37:37 dist-test-slave-drl0 krb5kdc[406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163857, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
I20260808 04:37:37.771064 2097 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:37:37.776607 2121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:37.778980 2097 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260808 04:37:37.779135 2097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "cf09a18f254848e9807ce04b266532ae"
format_stamp: "Formatted at 2026-08-08 04:37:36 on dist-test-slave-drl0"
server_key: "cc3efe768c65ce2847e6e58ca328ed9b"
server_key_iv: "b22bb7b85ba7776335fc0b12f876576d"
server_key_version: "kuduclusterkey@0"
I20260808 04:37:37.779997 2097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:37.808625 2097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:37.812629 2097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:37.812942 2097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:37.813764 2097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:37:37.815029 2097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:37:37.815142 2097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:37.815248 2097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:37:37.815311 2097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:37.826792 2097 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:34713
I20260808 04:37:37.832156 2097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:37:37.834857 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 2097
I20260808 04:37:37.835444 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.228.84:42901/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:39573
--local_ip_for_outbound_sockets=127.15.228.67
--tserver_master_addrs=127.15.228.126:34959
--webserver_port=37783
--webserver_interface=127.15.228.67
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:39253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.228.84:42901/kms with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:37:37.834077 2275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:34713 every 8 connection(s)
Aug 08 04:37:37 dist-test-slave-drl0 krb5kdc[406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163857, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:37:37.860226 2101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:37.839950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:47373 (local address 127.15.228.126:34959)
0808 04:37:37.840118 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:37.840124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:37.840703 (+ 579us) server_negotiation.cc:407] Connection header received
0808 04:37:37.842194 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:37.842199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:37.842289 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:37.842364 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:37.844404 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:37.845362 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:37.847435 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:37.847848 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:37.847941 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:37.854772 (+ 6831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:37.854779 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:37.854785 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:37.854825 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:37.856583 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:37.857652 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:37.857689 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:37.857694 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:37.857808 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:37.858376 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:37.858387 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:37.858391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:37.858716 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:37.858823 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:37.859758 (+ 935us) server_negotiation.cc:299] Negotiation successful
0808 04:37:37.859949 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:37:37.862277 2281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:37.838195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34959 (local address 127.15.228.66:47373)
0808 04:37:37.840502 (+ 2307us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:37.840565 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:37.841904 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:37.842880 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:37.842891 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:37.843360 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:37.844172 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:37.844195 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:37.845565 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:37.845569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:37.847251 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:37.847275 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:37.848199 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:37.848203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:37.848459 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:37.849655 (+ 1196us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:37.849682 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:37.851948 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:37.856748 (+ 4800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:37.856759 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:37.856787 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:37.857292 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:37.857957 (+ 665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:37.857963 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:37.857968 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:37.858216 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:37.858913 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:37.858945 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:37.859386 (+ 441us) client_negotiation.cc:769] Sending connection context
0808 04:37:37.860471 (+ 1085us) client_negotiation.cc:240] Negotiation successful
0808 04:37:37.860896 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2044,"thread_start_us":310,"threads_started":1}
I20260808 04:37:37.864264 2276 heartbeater.cc:347] Connected to a master server at 127.15.228.126:34959
I20260808 04:37:37.864584 2276 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:37.865325 2276 heartbeater.cc:511] Master 127.15.228.126:34959 requested a full tablet report, sending...
W20260808 04:37:37.867385 2276 heartbeater.cc:644] Failed to heartbeat to 127.15.228.126:34959 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 04:37:37.867522 2276 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:37.867704 2276 heartbeater.cc:511] Master 127.15.228.126:34959 requested a full tablet report, sending...
I20260808 04:37:37.868590 2276 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:37.868757 2276 heartbeater.cc:511] Master 127.15.228.126:34959 requested a full tablet report, sending...
W20260808 04:37:37.869540 2276 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:37:38.019585 2279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:38.020453 2279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:38.020635 2279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:38.027907 2279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:38.028023 2279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:38.028126 2279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:37:38.037616 2279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:39253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.228.84:42901/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:39573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=37783
--tserver_master_addrs=127.15.228.126:34959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:38.040220 2279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:38.042299 2279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:37:38.056604 2279 server_base.cc:1034] running on GCE node
W20260808 04:37:38.056720 2295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:38.057138 2293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:38.057303 2292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:38.057830 2279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:38.058961 2279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:38.060170 2279 hybrid_clock.cc:648] HybridClock initialized: now 1786163858060133 us; error 55 us; skew 500 ppm
Aug 08 04:37:38 dist-test-slave-drl0 krb5kdc[406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163858, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:38.064936 2279 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:37:38.066009 2279 webserver.cc:460] Webserver started at http://127.15.228.67:37783/ using document root <none> and password file <none>
I20260808 04:37:38.066777 2279 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:38.066907 2279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 04:37:38 dist-test-slave-drl0 krb5kdc[406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163858, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
I20260808 04:37:38.097823 2279 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:37:38.104331 2313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:38.106180 2279 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260808 04:37:38.106338 2279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "b606b0ddb1604441b7b0d07b4dcaea61"
format_stamp: "Formatted at 2026-08-08 04:37:36 on dist-test-slave-drl0"
server_key: "0cb957a58a7b5c8d9c0411b7c85d87f0"
server_key_iv: "816a1903a2f93a0fa9c0c2b692d0e49c"
server_key_version: "kuduclusterkey@0"
I20260808 04:37:38.107203 2279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:38.123049 2279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:38.127020 2279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:38.127337 2279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:38.128091 2279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:37:38.129177 2279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:37:38.129243 2279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:38.129298 2279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:37:38.129326 2279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:38.141906 2279 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:39573
I20260808 04:37:38.144433 2467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:39573 every 8 connection(s)
I20260808 04:37:38.145159 2279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:37:38.153468 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 2279
Aug 08 04:37:38 dist-test-slave-drl0 krb5kdc[406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163858, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:37:38.162510 2101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:38.147466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:52743 (local address 127.15.228.126:34959)
0808 04:37:38.147626 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:38.147630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:38.148746 (+ 1116us) server_negotiation.cc:407] Connection header received
0808 04:37:38.149720 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:38.149723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:38.149785 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:38.149861 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:38.150982 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:38.151542 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:38.152801 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:38.153043 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:38.153162 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:38.156634 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:38.156638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:38.156641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:38.156669 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:38.158783 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:38.159839 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:38.159844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:38.159846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:38.159960 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:38.160473 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:38.160477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:38.160479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:38.160796 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:38.160885 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:38.161973 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0808 04:37:38.162256 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:37:38.164287 2470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:38.148003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34959 (local address 127.15.228.67:52743)
0808 04:37:38.148589 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:38.148631 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:38.149505 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:38.150038 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:38.150044 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:38.150317 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:38.150802 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:38.150815 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:38.151684 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:38.151688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:38.152590 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:38.152600 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:38.153208 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:38.153212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:38.153391 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:38.154058 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:38.154082 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:38.156442 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:38.159049 (+ 2607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:38.159060 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:38.159088 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:38.159670 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:38.160091 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:38.160094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:38.160096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:38.160344 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:38.161004 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:38.161011 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:38.161563 (+ 552us) client_negotiation.cc:769] Sending connection context
0808 04:37:38.162755 (+ 1192us) client_negotiation.cc:240] Negotiation successful
0808 04:37:38.163234 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":153,"threads_started":1}
I20260808 04:37:38.165760 2468 heartbeater.cc:347] Connected to a master server at 127.15.228.126:34959
I20260808 04:37:38.166030 2468 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:38.166580 2468 heartbeater.cc:511] Master 127.15.228.126:34959 requested a full tablet report, sending...
W20260808 04:37:38.168331 2468 heartbeater.cc:644] Failed to heartbeat to 127.15.228.126:34959 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 04:37:38.168479 2468 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:38.168668 2468 heartbeater.cc:511] Master 127.15.228.126:34959 requested a full tablet report, sending...
I20260808 04:37:38.169482 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:38.154205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37476 (local address 127.15.228.126:34959)
0808 04:37:38.154888 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:38.154893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:38.154917 (+ 24us) server_negotiation.cc:407] Connection header received
0808 04:37:38.154983 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:38.154989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:38.155042 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:38.155126 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:38.156120 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:38.156691 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:38.157980 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:38.158360 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:38.158752 (+ 392us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:38.160501 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:38.160506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:38.160511 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:38.160581 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:38.162903 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:38.164220 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:38.164224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:38.164226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:38.164341 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:38.164983 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:38.164987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:38.164989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:38.165286 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:38.165526 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:38.165723 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 04:37:38.169285 (+ 3562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":131,"threads_started":1}
I20260808 04:37:38.169893 2468 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:38.170074 2468 heartbeater.cc:511] Master 127.15.228.126:34959 requested a full tablet report, sending...
W20260808 04:37:38.171100 2468 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 04:37:38.511487 2094 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:38.511847 2094 heartbeater.cc:511] Master 127.15.228.126:34959 requested a full tablet report, sending...
I20260808 04:37:38.513413 1808 ts_manager.cc:194] Registered new tserver with Master: ddf34562e21a49509ee8c375e7b357f4 (127.15.228.65:35951)
I20260808 04:37:38.515692 1808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:33311
Aug 08 04:37:38 dist-test-slave-drl0 krb5kdc[406](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786163858, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:38.869817 2276 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:38.870309 2276 heartbeater.cc:511] Master 127.15.228.126:34959 requested a full tablet report, sending...
I20260808 04:37:38.871459 1808 ts_manager.cc:194] Registered new tserver with Master: cf09a18f254848e9807ce04b266532ae (127.15.228.66:34713)
I20260808 04:37:38.872349 1808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:47373
Aug 08 04:37:39 dist-test-slave-drl0 krb5kdc[406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163858, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for HTTP/127.15.228.84@KRBTEST.COM
I20260808 04:37:39.171442 2468 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:39.171983 2468 heartbeater.cc:511] Master 127.15.228.126:34959 requested a full tablet report, sending...
I20260808 04:37:39.173653 1808 ts_manager.cc:194] Registered new tserver with Master: b606b0ddb1604441b7b0d07b4dcaea61 (127.15.228.67:39573)
I20260808 04:37:39.174764 1808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:52743
I20260808 04:37:39.178301 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:37:39.190553 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:39.180170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.15.228.126:34959)
0808 04:37:39.180442 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:39.180447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:39.180508 (+ 61us) server_negotiation.cc:407] Connection header received
0808 04:37:39.180662 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:39.180666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:39.180722 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:39.180826 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:39.181734 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.182585 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.183821 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.184183 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.184284 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:39.186318 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:39.186322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:39.186326 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:39.186404 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:39.188384 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:39.188898 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:39.188902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:39.188904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:39.189005 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:39.189347 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:39.189352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:39.189353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:39.189614 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:39.189771 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:39.190040 (+ 269us) server_negotiation.cc:299] Negotiation successful
0808 04:37:39.190189 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":104,"threads_started":1}
I20260808 04:37:39.195142 1808 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:37:39.200420 1808 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:37:39.226270 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:39.216495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:39573 (local address 127.0.0.1:41574)
0808 04:37:39.218569 (+ 2074us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:39.218585 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:39.218807 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:39.219582 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:39.219585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:39.219603 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:37:39.219698 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:39.219704 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.222160 (+ 2456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.222164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:39.223461 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:39.223473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.224718 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.224722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:39.224854 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:39.224921 (+ 67us) client_negotiation.cc:769] Sending connection context
0808 04:37:39.225916 (+ 995us) client_negotiation.cc:240] Negotiation successful
0808 04:37:39.226074 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1346,"thread_start_us":94,"threads_started":1}
I20260808 04:37:39.227176 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:39.216287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:35951 (local address 127.0.0.1:48078)
0808 04:37:39.217572 (+ 1285us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:39.217587 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:39.217700 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:39.218655 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:39.218663 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:39.218691 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:37:39.218860 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:39.218867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.220454 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.220458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:39.222270 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:39.222282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.226722 (+ 4440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.226727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:39.226828 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:39.226853 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:37:39.226953 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 04:37:39.227034 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1222,"thread_start_us":115,"threads_started":1}
I20260808 04:37:39.227746 2511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:39.216954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48078 (local address 127.15.228.65:35951)
0808 04:37:39.217431 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:39.217439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:39.217695 (+ 256us) server_negotiation.cc:407] Connection header received
0808 04:37:39.217980 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:39.217987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:39.218220 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:39.218376 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:37:39.219054 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.220309 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.222454 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.226321 (+ 3867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.226445 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:39.226648 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:39.227466 (+ 818us) server_negotiation.cc:299] Negotiation successful
0808 04:37:39.227592 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":104,"threads_started":1}
I20260808 04:37:39.228807 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:39.216936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.15.228.67:39573)
0808 04:37:39.219114 (+ 2178us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:39.219121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:39.219147 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:37:39.219227 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:39.219232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:39.219425 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:39.219851 (+ 426us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:37:39.220062 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.222025 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.223641 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.224570 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.227041 (+ 2471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:39.227194 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:39.227444 (+ 250us) server_negotiation.cc:299] Negotiation successful
0808 04:37:39.227561 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2060,"thread_start_us":81,"threads_started":1}
I20260808 04:37:39.231324 2005 tablet_service.cc:1467] Processing CreateTablet for tablet 183cac6cf1e546489898722c9a955cb0 (DEFAULT_TABLE table=test-table [id=cb2cec9a1594464ba9078f09311d246a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:37:39.231513 2373 tablet_service.cc:1467] Processing CreateTablet for tablet 183cac6cf1e546489898722c9a955cb0 (DEFAULT_TABLE table=test-table [id=cb2cec9a1594464ba9078f09311d246a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:37:39.232682 2005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 183cac6cf1e546489898722c9a955cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:37:39.232896 2373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 183cac6cf1e546489898722c9a955cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:37:39.234305 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:39.214683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:34713 (local address 127.0.0.1:36006)
0808 04:37:39.215274 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:39.215328 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:39.215465 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:39.227778 (+ 12313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:39.227781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:39.227804 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:37:39.227906 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:39.227914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.229485 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.229489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:39.232057 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:39.232070 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.233605 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.233609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:39.233719 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:39.233752 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 04:37:39.233809 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 04:37:39.233910 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":124,"threads_started":1}
I20260808 04:37:39.237408 2507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:39.214888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.15.228.66:34713)
0808 04:37:39.227188 (+ 12300us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:39.227195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:39.227227 (+ 32us) server_negotiation.cc:407] Connection header received
0808 04:37:39.227328 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:39.227334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:39.227594 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:39.228077 (+ 483us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:37:39.228262 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.229334 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.232236 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.233420 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.235056 (+ 1636us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:39.235237 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:39.235318 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 04:37:39.237178 (+ 1860us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12164,"thread_start_us":123,"threads_started":1}
I20260808 04:37:39.240842 2192 tablet_service.cc:1467] Processing CreateTablet for tablet 183cac6cf1e546489898722c9a955cb0 (DEFAULT_TABLE table=test-table [id=cb2cec9a1594464ba9078f09311d246a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:37:39.242060 2192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 183cac6cf1e546489898722c9a955cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:37:39.253048 2514 tablet_bootstrap.cc:492] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61: Bootstrap starting.
I20260808 04:37:39.253213 2512 tablet_bootstrap.cc:492] T 183cac6cf1e546489898722c9a955cb0 P ddf34562e21a49509ee8c375e7b357f4: Bootstrap starting.
I20260808 04:37:39.256255 2514 tablet_bootstrap.cc:654] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61: Neither blocks nor log segments found. Creating new log.
I20260808 04:37:39.256495 2512 tablet_bootstrap.cc:654] T 183cac6cf1e546489898722c9a955cb0 P ddf34562e21a49509ee8c375e7b357f4: Neither blocks nor log segments found. Creating new log.
I20260808 04:37:39.256879 2513 tablet_bootstrap.cc:492] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae: Bootstrap starting.
I20260808 04:37:39.257716 2512 log.cc:824] T 183cac6cf1e546489898722c9a955cb0 P ddf34562e21a49509ee8c375e7b357f4: Log is configured to *not* fsync() on all Append() calls
I20260808 04:37:39.259754 2514 log.cc:824] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61: Log is configured to *not* fsync() on all Append() calls
I20260808 04:37:39.260066 2513 tablet_bootstrap.cc:654] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae: Neither blocks nor log segments found. Creating new log.
I20260808 04:37:39.261053 2513 log.cc:824] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae: Log is configured to *not* fsync() on all Append() calls
I20260808 04:37:39.263123 2512 tablet_bootstrap.cc:492] T 183cac6cf1e546489898722c9a955cb0 P ddf34562e21a49509ee8c375e7b357f4: No bootstrap required, opened a new log
I20260808 04:37:39.263335 2512 ts_tablet_manager.cc:1397] T 183cac6cf1e546489898722c9a955cb0 P ddf34562e21a49509ee8c375e7b357f4: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260808 04:37:39.263525 2514 tablet_bootstrap.cc:492] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61: No bootstrap required, opened a new log
I20260808 04:37:39.263715 2514 ts_tablet_manager.cc:1397] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260808 04:37:39.267474 2512 raft_consensus.cc:348] T 183cac6cf1e546489898722c9a955cb0 P ddf34562e21a49509ee8c375e7b357f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09a18f254848e9807ce04b266532ae" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34713 } } peers { permanent_uuid: "ddf34562e21a49509ee8c375e7b357f4" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 35951 } } peers { permanent_uuid: "b606b0ddb1604441b7b0d07b4dcaea61" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39573 } }
I20260808 04:37:39.267768 2512 raft_consensus.cc:374] T 183cac6cf1e546489898722c9a955cb0 P ddf34562e21a49509ee8c375e7b357f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:37:39.267820 2512 raft_consensus.cc:729] T 183cac6cf1e546489898722c9a955cb0 P ddf34562e21a49509ee8c375e7b357f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf34562e21a49509ee8c375e7b357f4, State: Initialized, Role: FOLLOWER
I20260808 04:37:39.268304 2512 consensus_queue.cc:260] T 183cac6cf1e546489898722c9a955cb0 P ddf34562e21a49509ee8c375e7b357f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09a18f254848e9807ce04b266532ae" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34713 } } peers { permanent_uuid: "ddf34562e21a49509ee8c375e7b357f4" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 35951 } } peers { permanent_uuid: "b606b0ddb1604441b7b0d07b4dcaea61" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39573 } }
I20260808 04:37:39.268739 2514 raft_consensus.cc:348] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09a18f254848e9807ce04b266532ae" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34713 } } peers { permanent_uuid: "ddf34562e21a49509ee8c375e7b357f4" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 35951 } } peers { permanent_uuid: "b606b0ddb1604441b7b0d07b4dcaea61" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39573 } }
I20260808 04:37:39.269035 2514 raft_consensus.cc:374] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:37:39.269135 2514 raft_consensus.cc:729] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b606b0ddb1604441b7b0d07b4dcaea61, State: Initialized, Role: FOLLOWER
I20260808 04:37:39.269644 2514 consensus_queue.cc:260] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09a18f254848e9807ce04b266532ae" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34713 } } peers { permanent_uuid: "ddf34562e21a49509ee8c375e7b357f4" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 35951 } } peers { permanent_uuid: "b606b0ddb1604441b7b0d07b4dcaea61" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39573 } }
I20260808 04:37:39.269796 2512 ts_tablet_manager.cc:1428] T 183cac6cf1e546489898722c9a955cb0 P ddf34562e21a49509ee8c375e7b357f4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:37:39.270375 2094 heartbeater.cc:503] Master 127.15.228.126:34959 was elected leader, sending a full tablet report...
I20260808 04:37:39.270804 2514 ts_tablet_manager.cc:1428] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 04:37:39.272198 2468 heartbeater.cc:503] Master 127.15.228.126:34959 was elected leader, sending a full tablet report...
I20260808 04:37:39.275611 2513 tablet_bootstrap.cc:492] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae: No bootstrap required, opened a new log
I20260808 04:37:39.275892 2513 ts_tablet_manager.cc:1397] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20260808 04:37:39.281472 2513 raft_consensus.cc:348] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09a18f254848e9807ce04b266532ae" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34713 } } peers { permanent_uuid: "ddf34562e21a49509ee8c375e7b357f4" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 35951 } } peers { permanent_uuid: "b606b0ddb1604441b7b0d07b4dcaea61" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39573 } }
I20260808 04:37:39.281822 2513 raft_consensus.cc:374] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:37:39.281886 2513 raft_consensus.cc:729] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf09a18f254848e9807ce04b266532ae, State: Initialized, Role: FOLLOWER
I20260808 04:37:39.282516 2513 consensus_queue.cc:260] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09a18f254848e9807ce04b266532ae" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34713 } } peers { permanent_uuid: "ddf34562e21a49509ee8c375e7b357f4" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 35951 } } peers { permanent_uuid: "b606b0ddb1604441b7b0d07b4dcaea61" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39573 } }
I20260808 04:37:39.283602 2276 heartbeater.cc:503] Master 127.15.228.126:34959 was elected leader, sending a full tablet report...
I20260808 04:37:39.283895 2513 ts_tablet_manager.cc:1428] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260808 04:37:39.315145 2523 raft_consensus.cc:482] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:37:39.315448 2523 raft_consensus.cc:504] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09a18f254848e9807ce04b266532ae" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34713 } } peers { permanent_uuid: "ddf34562e21a49509ee8c375e7b357f4" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 35951 } } peers { permanent_uuid: "b606b0ddb1604441b7b0d07b4dcaea61" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39573 } }
I20260808 04:37:39.318672 2523 leader_election.cc:288] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddf34562e21a49509ee8c375e7b357f4 (127.15.228.65:35951), b606b0ddb1604441b7b0d07b4dcaea61 (127.15.228.67:39573)
I20260808 04:37:39.322645 2511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:39.317584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:34883 (local address 127.15.228.65:35951)
0808 04:37:39.317729 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:39.317734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:39.317756 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:37:39.317972 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:39.317976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:39.318037 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:39.318086 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:37:39.318576 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.319544 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.321246 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.321877 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.321985 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:39.322046 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:39.322411 (+ 365us) server_negotiation.cc:299] Negotiation successful
0808 04:37:39.322501 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:37:39.323091 2524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:39.317195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:35951 (local address 127.15.228.66:34883)
0808 04:37:39.317685 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:39.317702 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:39.317812 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:39.318244 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:39.318247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:39.318267 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:37:39.318436 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:39.318444 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.319705 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.319708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:39.321090 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:39.321102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.322147 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.322151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:39.322259 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:39.322284 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:37:39.322797 (+ 513us) client_negotiation.cc:240] Negotiation successful
0808 04:37:39.322892 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":146,"threads_started":1}
I20260808 04:37:39.323879 2030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "183cac6cf1e546489898722c9a955cb0" candidate_uuid: "cf09a18f254848e9807ce04b266532ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddf34562e21a49509ee8c375e7b357f4" is_pre_election: true
I20260808 04:37:39.324271 2030 raft_consensus.cc:2454] T 183cac6cf1e546489898722c9a955cb0 P ddf34562e21a49509ee8c375e7b357f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf09a18f254848e9807ce04b266532ae in term 0.
I20260808 04:37:39.324851 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:39.318512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:36811 (local address 127.15.228.67:39573)
0808 04:37:39.318634 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:39.318639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:39.319171 (+ 532us) server_negotiation.cc:407] Connection header received
0808 04:37:39.319378 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:39.319382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:39.319445 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:39.319503 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:37:39.319967 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.320721 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.322240 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.322860 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.322997 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:39.323046 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:39.324600 (+ 1554us) server_negotiation.cc:299] Negotiation successful
0808 04:37:39.324718 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:37:39.324855 2126 leader_election.cc:302] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf09a18f254848e9807ce04b266532ae, ddf34562e21a49509ee8c375e7b357f4; no voters:
I20260808 04:37:39.325207 2525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:39.318120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:39573 (local address 127.15.228.66:36811)
0808 04:37:39.319074 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:39.319088 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:39.319241 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:39.319652 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:39.319654 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:39.319673 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:37:39.319806 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:39.319813 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.320875 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.320879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:39.322085 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:39.322097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.324233 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.324237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:39.324341 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:39.324367 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 04:37:39.324990 (+ 623us) client_negotiation.cc:240] Negotiation successful
0808 04:37:39.325070 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"thread_start_us":98,"threads_started":1}
I20260808 04:37:39.325639 2523 raft_consensus.cc:2781] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:37:39.325737 2523 raft_consensus.cc:482] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:37:39.325791 2523 raft_consensus.cc:3037] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:37:39.325866 2395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "183cac6cf1e546489898722c9a955cb0" candidate_uuid: "cf09a18f254848e9807ce04b266532ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b606b0ddb1604441b7b0d07b4dcaea61" is_pre_election: true
I20260808 04:37:39.326175 2395 raft_consensus.cc:2454] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf09a18f254848e9807ce04b266532ae in term 0.
I20260808 04:37:39.327395 2523 raft_consensus.cc:504] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09a18f254848e9807ce04b266532ae" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34713 } } peers { permanent_uuid: "ddf34562e21a49509ee8c375e7b357f4" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 35951 } } peers { permanent_uuid: "b606b0ddb1604441b7b0d07b4dcaea61" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39573 } }
I20260808 04:37:39.328169 2395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "183cac6cf1e546489898722c9a955cb0" candidate_uuid: "cf09a18f254848e9807ce04b266532ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b606b0ddb1604441b7b0d07b4dcaea61"
I20260808 04:37:39.328305 2395 raft_consensus.cc:3037] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:37:39.329785 2395 raft_consensus.cc:2454] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf09a18f254848e9807ce04b266532ae in term 1.
I20260808 04:37:39.330092 2030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "183cac6cf1e546489898722c9a955cb0" candidate_uuid: "cf09a18f254848e9807ce04b266532ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddf34562e21a49509ee8c375e7b357f4"
I20260808 04:37:39.330171 2123 leader_election.cc:302] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b606b0ddb1604441b7b0d07b4dcaea61, cf09a18f254848e9807ce04b266532ae; no voters:
I20260808 04:37:39.330235 2030 raft_consensus.cc:3037] T 183cac6cf1e546489898722c9a955cb0 P ddf34562e21a49509ee8c375e7b357f4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:37:39.330533 2523 leader_election.cc:288] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [CANDIDATE]: Term 1 election: Requested vote from peers ddf34562e21a49509ee8c375e7b357f4 (127.15.228.65:35951), b606b0ddb1604441b7b0d07b4dcaea61 (127.15.228.67:39573)
I20260808 04:37:39.330672 2523 raft_consensus.cc:2781] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:37:39.330771 2523 raft_consensus.cc:686] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [term 1 LEADER]: Becoming Leader. State: Replica: cf09a18f254848e9807ce04b266532ae, State: Running, Role: LEADER
I20260808 04:37:39.331185 2523 consensus_queue.cc:237] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09a18f254848e9807ce04b266532ae" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34713 } } peers { permanent_uuid: "ddf34562e21a49509ee8c375e7b357f4" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 35951 } } peers { permanent_uuid: "b606b0ddb1604441b7b0d07b4dcaea61" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39573 } }
I20260808 04:37:39.331679 2030 raft_consensus.cc:2454] T 183cac6cf1e546489898722c9a955cb0 P ddf34562e21a49509ee8c375e7b357f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf09a18f254848e9807ce04b266532ae in term 1.
I20260808 04:37:39.336010 1818 catalog_manager.cc:5495] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae reported cstate change: term changed from 0 to 1, leader changed from <none> to cf09a18f254848e9807ce04b266532ae (127.15.228.66). New cstate: current_term: 1 leader_uuid: "cf09a18f254848e9807ce04b266532ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09a18f254848e9807ce04b266532ae" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddf34562e21a49509ee8c375e7b357f4" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 35951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b606b0ddb1604441b7b0d07b4dcaea61" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39573 } health_report { overall_health: UNKNOWN } } }
W20260808 04:37:39.336489 2277 tablet.cc:2369] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:37:39.352661 2507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:39.347770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36010 (local address 127.15.228.66:34713)
0808 04:37:39.347896 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:39.347901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:39.347916 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:37:39.348113 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:39.348116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:39.348189 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:39.348312 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:37:39.348579 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.349432 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.350646 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:39.351027 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:39.351144 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:39.351554 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:37:39.352164 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:37:39.352300 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:39.352416 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 04:37:39.352483 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 04:37:39.361773 2395 raft_consensus.cc:1261] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61 [term 1 FOLLOWER]: Refusing update from remote peer cf09a18f254848e9807ce04b266532ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:37:39.362953 2030 raft_consensus.cc:1261] T 183cac6cf1e546489898722c9a955cb0 P ddf34562e21a49509ee8c375e7b357f4 [term 1 FOLLOWER]: Refusing update from remote peer cf09a18f254848e9807ce04b266532ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:37:39.364261 2523 consensus_queue.cc:1035] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b606b0ddb1604441b7b0d07b4dcaea61" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:37:39.364799 2523 consensus_queue.cc:1035] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddf34562e21a49509ee8c375e7b357f4" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 35951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:37:39.380612 2530 mvcc.cc:205] Tried to move back new op lower bound from 7316127167932547072 to 7316127167825485824. Current Snapshot: MvccSnapshot[applied={T|T < 7316127167932547072}]
I20260808 04:37:39.391098 2535 mvcc.cc:205] Tried to move back new op lower bound from 7316127167932547072 to 7316127167825485824. Current Snapshot: MvccSnapshot[applied={T|T < 7316127167932547072}]
I20260808 04:37:39.393671 2531 mvcc.cc:205] Tried to move back new op lower bound from 7316127167932547072 to 7316127167825485824. Current Snapshot: MvccSnapshot[applied={T|T < 7316127167932547072}]
I20260808 04:37:39.397044 1815 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:37:39.397256 1815 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:37:39.402733 1815 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aad7a100f6844835bc4641e9ef8c9a63: Sending DeleteTablet for 3 replicas of tablet 183cac6cf1e546489898722c9a955cb0
I20260808 04:37:39.404246 2192 tablet_service.cc:1514] Processing DeleteTablet for tablet 183cac6cf1e546489898722c9a955cb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:37:39 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:36006
I20260808 04:37:39.405879 2005 tablet_service.cc:1514] Processing DeleteTablet for tablet 183cac6cf1e546489898722c9a955cb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:37:39 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:48078
I20260808 04:37:39.407523 2373 tablet_service.cc:1514] Processing DeleteTablet for tablet 183cac6cf1e546489898722c9a955cb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:37:39 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:41574
I20260808 04:37:39.409387 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 1871
I20260808 04:37:39.411929 2545 tablet_replica.cc:330] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae: stopping tablet replica
I20260808 04:37:39.412622 2545 raft_consensus.cc:2229] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:37:39.414242 2545 raft_consensus.cc:2258] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:37:39.420814 2545 ts_tablet_manager.cc:1905] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:37:39.421787 2547 tablet_replica.cc:330] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61: stopping tablet replica
I20260808 04:37:39.423702 2547 raft_consensus.cc:2229] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:37:39.426862 2547 raft_consensus.cc:2258] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:37:39.430625 2545 ts_tablet_manager.cc:1918] T 183cac6cf1e546489898722c9a955cb0 P cf09a18f254848e9807ce04b266532ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:37:39.432878 2547 ts_tablet_manager.cc:1905] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 04:37:39.435974 1796 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.228.65:35951 (error 108)
I20260808 04:37:39.437927 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 2097
I20260808 04:37:39.458102 2547 ts_tablet_manager.cc:1918] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:37:39.460002 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:39.459503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:35951 (local address 127.0.0.1:48084)
0808 04:37:39.459782 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:39.459884 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.65:35951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
W20260808 04:37:39.460292 1796 catalog_manager.cc:4584] TS ddf34562e21a49509ee8c375e7b357f4 (127.15.228.65:35951): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 183cac6cf1e546489898722c9a955cb0: Network error: Client connection negotiation failed: client connection to 127.15.228.65:35951: connect: Connection refused (error 111)
I20260808 04:37:39.460687 2547 log.cc:1196] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/183cac6cf1e546489898722c9a955cb0
I20260808 04:37:39.461258 2547 ts_tablet_manager.cc:1939] T 183cac6cf1e546489898722c9a955cb0 P b606b0ddb1604441b7b0d07b4dcaea61: Deleting consensus metadata
I20260808 04:37:39.463568 1793 catalog_manager.cc:4841] TS b606b0ddb1604441b7b0d07b4dcaea61 (127.15.228.67:39573): tablet 183cac6cf1e546489898722c9a955cb0 (table test-table [id=cb2cec9a1594464ba9078f09311d246a]) successfully deleted
I20260808 04:37:39.475324 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 2279
I20260808 04:37:39.477843 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:39.475391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:34713 (local address 127.0.0.1:36026)
0808 04:37:39.477661 (+ 2270us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:39.477735 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.66:34713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2164}
I20260808 04:37:39.491904 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 1775
I20260808 04:37:39.505511 16273 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 04:37:39.864127 16273 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 04:37:39.864266 16273 mini_ranger.cc:66] Stopping Ranger...
I20260808 04:37:40.231554 16273 mini_ranger.cc:68] Stopped Ranger
2026-08-08 04:37:40.231 UTC [445] LOG: received smart shutdown request
2026-08-08 04:37:40.237 UTC [445] LOG: background worker "logical replication launcher" (PID 462) exited with exit code 1
2026-08-08 04:37:40.237 UTC [453] LOG: shutting down
2026-08-08 04:37:40.459 UTC [445] LOG: database system is shut down
2026-08-08T04:37:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (59276 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:37:40 dist-test-slave-drl0 krb5kdc[2560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:37:40 dist-test-slave-drl0 krb5kdc[2560](info): set up 2 sockets
Aug 08 04:37:40 dist-test-slave-drl0 krb5kdc[2560](info): commencing operation
krb5kdc: starting...
W20260808 04:37:41.928463 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.289s user 0.002s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:37:42 dist-test-slave-drl0 krb5kdc[2560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:37:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:37:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:37:42.091521 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:39663
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:35289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:39663,127.15.228.125:46545,127.15.228.124:46219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:37:42.229614 2585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:42.229971 2585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:42.230049 2585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:42.234256 2585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:37:42.234364 2585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:42.234401 2585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:42.234439 2585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:37:42.234490 2585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:37:42.240136 2585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:35289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:39663,127.15.228.125:46545,127.15.228.124:46219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:39663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:42.241412 2585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:42.242525 2585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:42.253221 2594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:42.253214 2592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:42.253206 2591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:42.253816 2585 server_base.cc:1034] running on GCE node
I20260808 04:37:42.254454 2585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:42.255720 2585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:42.256950 2585 hybrid_clock.cc:648] HybridClock initialized: now 1786163862256916 us; error 66 us; skew 500 ppm
Aug 08 04:37:42 dist-test-slave-drl0 krb5kdc[2560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163862, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:42.263566 2585 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:37:42.264561 2585 webserver.cc:460] Webserver started at http://127.15.228.126:36773/ using document root <none> and password file <none>
I20260808 04:37:42.265130 2585 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:42.265192 2585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:37:42.265388 2585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:37:42.268246 2585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "5605e07666bd45f3877473ed2a562cef"
format_stamp: "Formatted at 2026-08-08 04:37:42 on dist-test-slave-drl0"
server_key: "162725247530d1568574f4f2d674d811"
server_key_iv: "fe7bc9b2cfb13bdb0ece308fe2976762"
server_key_version: "encryptionkey@0"
I20260808 04:37:42.269106 2585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "5605e07666bd45f3877473ed2a562cef"
format_stamp: "Formatted at 2026-08-08 04:37:42 on dist-test-slave-drl0"
server_key: "162725247530d1568574f4f2d674d811"
server_key_iv: "fe7bc9b2cfb13bdb0ece308fe2976762"
server_key_version: "encryptionkey@0"
I20260808 04:37:42.274802 2585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 04:37:42.278090 2604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:42.279788 2585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:37:42.279914 2585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "5605e07666bd45f3877473ed2a562cef"
format_stamp: "Formatted at 2026-08-08 04:37:42 on dist-test-slave-drl0"
server_key: "162725247530d1568574f4f2d674d811"
server_key_iv: "fe7bc9b2cfb13bdb0ece308fe2976762"
server_key_version: "encryptionkey@0"
I20260808 04:37:42.280083 2585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:42.295564 2585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:42.299732 2585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:42.300122 2585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:42.310367 2585 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:39663
I20260808 04:37:42.310745 2682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:39663 every 8 connection(s)
I20260808 04:37:42.311932 2585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:37:42.312460 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 2585
I20260808 04:37:42.312633 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:37:42.312989 16273 external_mini_cluster.cc:1428] Setting key 3c0d0f0e5f1afb7caf5eded8fc5ef23b
I20260808 04:37:42.315569 2685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:37:42.319617 2685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.228.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.125@KRBTEST.COM" created.
Aug 08 04:37:42 dist-test-slave-drl0 krb5kdc[2560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163862, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:37:42.339586 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.322073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:39663 (local address 127.0.0.1:40882)
0808 04:37:42.322806 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.322868 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:42.323838 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:42.324683 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:42.324707 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:42.325169 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:42.325447 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:42.325462 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.327464 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.327469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:42.328705 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:42.328716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.329630 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.329634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:42.329782 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.330568 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:42.330595 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:42.332861 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:42.335712 (+ 2851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:42.335722 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:42.335731 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:42.335975 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:42.336506 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:42.336511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:42.336512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:42.336779 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:42.337696 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:42.337707 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:42.337923 (+ 216us) client_negotiation.cc:769] Sending connection context
0808 04:37:42.338169 (+ 246us) client_negotiation.cc:240] Negotiation successful
0808 04:37:42.338474 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":1,"thread_start_us":153,"threads_started":1}
I20260808 04:37:42.339617 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.323009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40882 (local address 127.15.228.126:39663)
0808 04:37:42.323476 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:42.323499 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:42.323581 (+ 82us) server_negotiation.cc:407] Connection header received
0808 04:37:42.324203 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:42.324233 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:42.324450 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:42.324618 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:42.325627 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.327291 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.328928 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.329448 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.329647 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.333069 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:42.333084 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:42.333099 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:42.333140 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:42.335548 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.336146 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.336155 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.336160 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.336292 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.336990 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.336998 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.336999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.337449 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:42.337664 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:42.338412 (+ 748us) server_negotiation.cc:299] Negotiation successful
0808 04:37:42.338653 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":179,"thread_start_us":99,"threads_started":1}
I20260808 04:37:42.342347 2685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:37:42.343324 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.342972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:46545 (local address 127.0.0.1:56612)
0808 04:37:42.343153 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.343249 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:46545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260808 04:37:42.343549 2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.228.125:46545: connect: Connection refused (error 111)
I20260808 04:37:42.344539 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.344281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:46545 (local address 127.0.0.1:56622)
0808 04:37:42.344414 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.344483 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:46545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260808 04:37:42.344735 2685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.125:46545: Network error: Client connection negotiation failed: client connection to 127.15.228.125:46545: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.125.keytab.
Entry for principal kudu/127.15.228.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.125.keytab.
I20260808 04:37:42.347430 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.125:46545
--webserver_interface=127.15.228.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.125.keytab
--principal=kudu/127.15.228.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:35289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:39663,127.15.228.125:46545,127.15.228.124:46219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:37:42.394021 2685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } attempt: 1
I20260808 04:37:42.395058 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.394658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:46545 (local address 127.0.0.1:56636)
0808 04:37:42.394847 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.394915 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:46545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 04:37:42.396350 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.396011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:46545 (local address 127.0.0.1:56650)
0808 04:37:42.396211 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.396294 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:46545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260808 04:37:42.396689 2685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.125:46545: Network error: Client connection negotiation failed: client connection to 127.15.228.125:46545: connect: Connection refused (error 111)
I20260808 04:37:42.464928 2685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } attempt: 2
I20260808 04:37:42.466383 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.465871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:46545 (local address 127.0.0.1:56656)
0808 04:37:42.466113 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.466183 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:46545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260808 04:37:42.467727 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.467384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:46545 (local address 127.0.0.1:56670)
0808 04:37:42.467542 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.467619 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:46545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260808 04:37:42.467896 2685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.125:46545: Network error: Client connection negotiation failed: client connection to 127.15.228.125:46545: connect: Connection refused (error 111)
W20260808 04:37:42.477418 2693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:42.477907 2693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:42.478011 2693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:42.483911 2693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:37:42.484021 2693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:42.484061 2693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:42.484120 2693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:37:42.484177 2693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:37:42.490449 2693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:35289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:39663,127.15.228.125:46545,127.15.228.124:46219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.125.keytab
--principal=kudu/127.15.228.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.125:46545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.228.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:42.491746 2693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:42.492842 2693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:42.504277 2701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:42.504377 2693 server_base.cc:1034] running on GCE node
W20260808 04:37:42.504359 2702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:42.504520 2704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:42.505178 2693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:42.506537 2693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:42.507792 2693 hybrid_clock.cc:648] HybridClock initialized: now 1786163862507765 us; error 36 us; skew 500 ppm
Aug 08 04:37:42 dist-test-slave-drl0 krb5kdc[2560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163862, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:42.511505 2693 init.cc:377] Logged in from keytab as kudu/127.15.228.125@KRBTEST.COM (short username kudu)
I20260808 04:37:42.512900 2693 webserver.cc:460] Webserver started at http://127.15.228.125:40641/ using document root <none> and password file <none>
I20260808 04:37:42.513576 2693 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:42.513688 2693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:37:42.513957 2693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:37:42.516440 2693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-1/data/instance:
uuid: "ad79d179d9604f34afe73e6aca8f0fd2"
format_stamp: "Formatted at 2026-08-08 04:37:42 on dist-test-slave-drl0"
server_key: "f16f5ed8facd2a8b72a901cd91993b8a"
server_key_iv: "e4bd05d7f4d5989668fd1c1b40ccc27e"
server_key_version: "encryptionkey@0"
I20260808 04:37:42.517091 2693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-1/wal/instance:
uuid: "ad79d179d9604f34afe73e6aca8f0fd2"
format_stamp: "Formatted at 2026-08-08 04:37:42 on dist-test-slave-drl0"
server_key: "f16f5ed8facd2a8b72a901cd91993b8a"
server_key_iv: "e4bd05d7f4d5989668fd1c1b40ccc27e"
server_key_version: "encryptionkey@0"
I20260808 04:37:42.521579 2693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:37:42.524323 2715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:42.525895 2693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:37:42.526079 2693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-1/wal
uuid: "ad79d179d9604f34afe73e6aca8f0fd2"
format_stamp: "Formatted at 2026-08-08 04:37:42 on dist-test-slave-drl0"
server_key: "f16f5ed8facd2a8b72a901cd91993b8a"
server_key_iv: "e4bd05d7f4d5989668fd1c1b40ccc27e"
server_key_version: "encryptionkey@0"
I20260808 04:37:42.526249 2693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:42.538731 2693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:42.542689 2693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:42.542985 2693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:42.550707 2693 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.125:46545
I20260808 04:37:42.551255 2791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.125:46545 every 8 connection(s)
I20260808 04:37:42.551941 2693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-1/data/info.pb
I20260808 04:37:42.555289 2792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:37:42.556890 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 2693
I20260808 04:37:42.557044 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-1/wal/instance
I20260808 04:37:42.557411 16273 external_mini_cluster.cc:1428] Setting key db4574f2d0e700a158832be7bbb311a0
I20260808 04:37:42.559119 2685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } attempt: 3
I20260808 04:37:42.561118 2792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.228.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.124@KRBTEST.COM" created.
Aug 08 04:37:42 dist-test-slave-drl0 krb5kdc[2560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163862, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for kudu/127.15.228.125@KRBTEST.COM
Aug 08 04:37:42 dist-test-slave-drl0 krb5kdc[2560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163862, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.125@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:37:42.579169 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.559938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:46545 (local address 127.0.0.1:56678)
0808 04:37:42.560237 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.560258 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:42.560374 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:42.563683 (+ 3309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:42.563689 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:42.563936 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:42.564127 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:42.564135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.566546 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.566552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:42.567327 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:42.567339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.568381 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.568387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:42.568503 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.569106 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:42.569127 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:42.572973 (+ 3846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:42.576705 (+ 3732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:42.576710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:42.576714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:42.577012 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:42.577540 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:42.577546 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:42.577551 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:42.577663 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:42.578549 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:42.578551 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:42.578647 (+ 96us) client_negotiation.cc:769] Sending connection context
0808 04:37:42.578778 (+ 131us) client_negotiation.cc:240] Negotiation successful
0808 04:37:42.578978 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260808 04:37:42.579682 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.563107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.15.228.126:39663)
0808 04:37:42.563262 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:42.563269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:42.563684 (+ 415us) server_negotiation.cc:407] Connection header received
0808 04:37:42.564232 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:42.564236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:42.564312 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:42.564369 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:42.565865 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.566741 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.568063 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.568449 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.568551 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.573988 (+ 5437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:42.573995 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:42.574000 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:42.574041 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:42.576680 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.577593 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.577596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.577597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.577682 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.578135 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.578139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.578140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.578362 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:42.578479 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:42.579359 (+ 880us) server_negotiation.cc:299] Negotiation successful
0808 04:37:42.579502 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 04:37:42.580511 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.563088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:39663 (local address 127.0.0.1:40898)
0808 04:37:42.563552 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.563604 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:42.563951 (+ 347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:42.564649 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:42.564655 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:42.565276 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:42.565607 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:42.565651 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.566893 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.566897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:42.567929 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:42.567939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.568629 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.568634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:42.568724 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.569417 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:42.569443 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:42.573788 (+ 4345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:42.576877 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:42.576891 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:42.576906 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:42.577417 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:42.577871 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:42.577877 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:42.577882 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:42.578002 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:42.578499 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:42.578510 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:42.578741 (+ 231us) client_negotiation.cc:769] Sending connection context
0808 04:37:42.578956 (+ 215us) client_negotiation.cc:240] Negotiation successful
0808 04:37:42.579390 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":114,"thread_start_us":189,"threads_started":1}
I20260808 04:37:42.581060 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.560851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.15.228.125:46545)
0808 04:37:42.561519 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:42.561543 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:42.561593 (+ 50us) server_negotiation.cc:407] Connection header received
0808 04:37:42.562746 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:42.562775 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:42.563315 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:42.564356 (+ 1041us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:42.564959 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.566415 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.567485 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.568238 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.568454 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.573311 (+ 4857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:42.573323 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:42.573347 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:42.573395 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:42.576532 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.577242 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.577257 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.577259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.577417 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.577785 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.577789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.577792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.578458 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:42.578592 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:42.578905 (+ 313us) server_negotiation.cc:299] Negotiation successful
0808 04:37:42.580860 (+ 1955us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":473,"thread_start_us":179,"threads_started":1}
I20260808 04:37:42.582717 2792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:37:42.583487 2685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:37:42.584766 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.584352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:46219 (local address 127.0.0.1:46840)
0808 04:37:42.584562 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.584648 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 04:37:42.585817 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.585517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:46219 (local address 127.0.0.1:46844)
0808 04:37:42.585693 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.585743 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260808 04:37:42.586140 2685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.124:46219: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
Aug 08 04:37:42 dist-test-slave-drl0 krb5kdc[2560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163862, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.125@KRBTEST.COM for kudu/127.15.228.125@KRBTEST.COM
I20260808 04:37:42.594539 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.583606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:46545 (local address 127.0.0.1:56680)
0808 04:37:42.583716 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.583731 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:42.583824 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:42.584276 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:42.584280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:42.584684 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:42.584835 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:42.584845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.586093 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.586096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:42.586739 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:42.586749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.587598 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.587602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:42.587717 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.588290 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:42.588302 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:42.590216 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:42.592745 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:42.592749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:42.592752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:42.592976 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:42.593301 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:42.593308 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:42.593310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:42.593387 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:42.593828 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:42.593831 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:42.593927 (+ 96us) client_negotiation.cc:769] Sending connection context
0808 04:37:42.594040 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 04:37:42.594362 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":136}
I20260808 04:37:42.594539 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.583794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.15.228.125:46545)
0808 04:37:42.583926 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:42.583930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:42.583942 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:37:42.583979 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:42.583981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:42.584013 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:42.584270 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:42.585412 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.585981 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.586873 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.587440 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.587546 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.590449 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:42.590453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:42.590455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:42.590482 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:42.592586 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.593107 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.593111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.593115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.593198 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.593485 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.593489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.593491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.593727 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:42.593844 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:42.594100 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 04:37:42.594306 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":65}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.124.keytab.
Entry for principal kudu/127.15.228.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.124.keytab.
I20260808 04:37:42.595973 2792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:37:42.596107 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.124:46219
--webserver_interface=127.15.228.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.124.keytab
--principal=kudu/127.15.228.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:35289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:39663,127.15.228.125:46545,127.15.228.124:46219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:37:42.597016 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.596580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:46219 (local address 127.0.0.1:46852)
0808 04:37:42.596783 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.596916 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260808 04:37:42.598064 2720 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
I20260808 04:37:42.599244 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.598917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:46219 (local address 127.0.0.1:46864)
0808 04:37:42.599099 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.599160 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260808 04:37:42.599439 2792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.124:46219: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
I20260808 04:37:42.617360 2685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } attempt: 1
I20260808 04:37:42.618476 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.618080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:46219 (local address 127.0.0.1:46872)
0808 04:37:42.618283 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.618372 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260808 04:37:42.619359 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.619089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:46219 (local address 127.0.0.1:46886)
0808 04:37:42.619225 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.619287 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 04:37:42.619560 2685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.124:46219: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
I20260808 04:37:42.648731 2792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } attempt: 1
I20260808 04:37:42.649853 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.649437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:46219 (local address 127.0.0.1:46890)
0808 04:37:42.649641 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.649744 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 04:37:42.651353 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.651101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:46219 (local address 127.0.0.1:46902)
0808 04:37:42.651220 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.651279 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 04:37:42.651528 2792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.124:46219: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
I20260808 04:37:42.694799 2685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } attempt: 2
I20260808 04:37:42.696000 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.695621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:46219 (local address 127.0.0.1:46914)
0808 04:37:42.695856 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.695916 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260808 04:37:42.696856 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.696638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:46219 (local address 127.0.0.1:46918)
0808 04:37:42.696749 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.696804 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 04:37:42.696979 2685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.124:46219: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
I20260808 04:37:42.719734 2792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } attempt: 2
I20260808 04:37:42.720711 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.720399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:46219 (local address 127.0.0.1:46922)
0808 04:37:42.720572 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.720634 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 04:37:42.721769 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.721263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:46219 (local address 127.0.0.1:46932)
0808 04:37:42.721533 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.721657 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 04:37:42.721967 2792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.124:46219: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
W20260808 04:37:42.743450 2799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:42.743911 2799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:42.744118 2799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:42.749711 2799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:37:42.749858 2799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:42.749905 2799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:42.749970 2799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:37:42.750011 2799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:37:42.755991 2799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:35289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:39663,127.15.228.125:46545,127.15.228.124:46219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.124.keytab
--principal=kudu/127.15.228.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.124:46219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.228.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:42.757256 2799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:42.758524 2799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:42.769279 2812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:42.769266 2809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:42.769294 2808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:42.769747 2799 server_base.cc:1034] running on GCE node
I20260808 04:37:42.770859 2799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:42.772830 2799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:42.774132 2799 hybrid_clock.cc:648] HybridClock initialized: now 1786163862774086 us; error 72 us; skew 500 ppm
Aug 08 04:37:42 dist-test-slave-drl0 krb5kdc[2560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163862, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:42.778394 2799 init.cc:377] Logged in from keytab as kudu/127.15.228.124@KRBTEST.COM (short username kudu)
I20260808 04:37:42.779584 2799 webserver.cc:460] Webserver started at http://127.15.228.124:36055/ using document root <none> and password file <none>
I20260808 04:37:42.780256 2799 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:42.780335 2799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:37:42.780596 2799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:37:42.782986 2799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-2/data/instance:
uuid: "f2580cee4b2f4f11a4ce89730cab66dc"
format_stamp: "Formatted at 2026-08-08 04:37:42 on dist-test-slave-drl0"
server_key: "d50cd31ecf5d6a700c2ba6d389776d67"
server_key_iv: "733dca87b05b42b55e51b5dc807bddde"
server_key_version: "encryptionkey@0"
I20260808 04:37:42.783671 2799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-2/wal/instance:
uuid: "f2580cee4b2f4f11a4ce89730cab66dc"
format_stamp: "Formatted at 2026-08-08 04:37:42 on dist-test-slave-drl0"
server_key: "d50cd31ecf5d6a700c2ba6d389776d67"
server_key_iv: "733dca87b05b42b55e51b5dc807bddde"
server_key_version: "encryptionkey@0"
I20260808 04:37:42.789178 2799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:37:42.792480 2820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:42.794222 2799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:37:42.794415 2799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-2/wal
uuid: "f2580cee4b2f4f11a4ce89730cab66dc"
format_stamp: "Formatted at 2026-08-08 04:37:42 on dist-test-slave-drl0"
server_key: "d50cd31ecf5d6a700c2ba6d389776d67"
server_key_iv: "733dca87b05b42b55e51b5dc807bddde"
server_key_version: "encryptionkey@0"
I20260808 04:37:42.794559 2799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:42.796164 2685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } attempt: 3
I20260808 04:37:42.797142 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.796720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:46219 (local address 127.0.0.1:46938)
0808 04:37:42.796990 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.797058 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260808 04:37:42.798036 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.797806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:46219 (local address 127.0.0.1:46954)
0808 04:37:42.797930 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.797980 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260808 04:37:42.798233 2685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.124:46219: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
I20260808 04:37:42.810901 2799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:42.813236 2792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } attempt: 3
I20260808 04:37:42.814782 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.814162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:46219 (local address 127.0.0.1:46966)
0808 04:37:42.814505 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.814610 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260808 04:37:42.816332 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.815925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:46219 (local address 127.0.0.1:46982)
0808 04:37:42.816148 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.816213 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260808 04:37:42.816566 2792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.124:46219: Network error: Client connection negotiation failed: client connection to 127.15.228.124:46219: connect: Connection refused (error 111)
I20260808 04:37:42.816618 2799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:42.816838 2799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:42.826186 2799 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.124:46219
I20260808 04:37:42.826521 2892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.124:46219 every 8 connection(s)
I20260808 04:37:42.827728 2799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-2/data/info.pb
I20260808 04:37:42.831965 2894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:37:42.837184 2894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:37:42.837987 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 2799
I20260808 04:37:42.838147 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786163668922094-16273-0/minicluster-data/master-2/wal/instance
I20260808 04:37:42.838495 16273 external_mini_cluster.cc:1428] Setting key ff26f934e577405a26018cf9a35d474d
Aug 08 04:37:42 dist-test-slave-drl0 krb5kdc[2560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163862, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.124@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
Aug 08 04:37:42 dist-test-slave-drl0 krb5kdc[2560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.124@KRBTEST.COM
I20260808 04:37:42.854796 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.841241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40912 (local address 127.15.228.126:39663)
0808 04:37:42.841388 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:42.841395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:42.841963 (+ 568us) server_negotiation.cc:407] Connection header received
0808 04:37:42.842566 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:42.842573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:42.842631 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:42.842757 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:42.844110 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.844695 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.845994 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.846391 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.846496 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.850301 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:42.850305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:42.850308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:42.850349 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:42.852338 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.853053 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.853056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.853058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.853150 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.853523 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.853528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.853530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.853741 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:42.853855 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:42.854454 (+ 599us) server_negotiation.cc:299] Negotiation successful
0808 04:37:42.854601 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:37:42.855763 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.841346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:39663 (local address 127.0.0.1:40912)
0808 04:37:42.841807 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.841862 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:42.842158 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:42.842818 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:42.842824 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:42.843448 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:42.843899 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:42.843922 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.844825 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.844831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:42.845837 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:42.845848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.846563 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.846569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:42.846826 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.847684 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:42.847709 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:42.850030 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:42.852581 (+ 2551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:42.852594 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:42.852607 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:42.852926 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:42.853252 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:42.853256 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:42.853258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:42.853418 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:42.853856 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:42.853864 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:42.854163 (+ 299us) client_negotiation.cc:769] Sending connection context
0808 04:37:42.854371 (+ 208us) client_negotiation.cc:240] Negotiation successful
0808 04:37:42.854749 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":189,"threads_started":1}
I20260808 04:37:42.856684 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.839986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46996 (local address 127.15.228.124:46219)
0808 04:37:42.840555 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:42.840569 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:42.840603 (+ 34us) server_negotiation.cc:407] Connection header received
0808 04:37:42.841691 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:42.841724 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:42.842176 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:42.842593 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:42.844441 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.846484 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.847911 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.848498 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.848678 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.852176 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:42.852186 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:42.852205 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:42.852247 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:42.854283 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.854900 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.854909 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.854911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.855054 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.855392 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.855396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.855398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.855729 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:42.855985 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:42.856204 (+ 219us) server_negotiation.cc:299] Negotiation successful
0808 04:37:42.856385 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":201,"threads_started":1}
I20260808 04:37:42.857604 2894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:37:42 dist-test-slave-drl0 krb5kdc[2560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163862, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.124@KRBTEST.COM for kudu/127.15.228.125@KRBTEST.COM
Aug 08 04:37:42 dist-test-slave-drl0 krb5kdc[2560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:37:42.867671 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.858422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:46545 (local address 127.0.0.1:56684)
0808 04:37:42.858589 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.858608 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:42.858695 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:42.859185 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:42.859188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:42.859400 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:42.859560 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:42.859570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.860573 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.860577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:42.861319 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:42.861328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.861830 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.861834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:42.861945 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.862590 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:42.862603 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:42.863991 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:42.866026 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:42.866029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:42.866032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:42.866240 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:42.866542 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:42.866546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:42.866548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:42.866633 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:42.867100 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:42.867103 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:42.867209 (+ 106us) client_negotiation.cc:769] Sending connection context
0808 04:37:42.867331 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 04:37:42.867487 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 04:37:42.867671 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.858570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.15.228.125:46545)
0808 04:37:42.858815 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:42.858819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:42.858837 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:37:42.858883 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:42.858885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:42.859030 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:42.859159 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:42.859725 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.860413 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.861442 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.861688 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.861824 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.864156 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:42.864159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:42.864161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:42.864185 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:42.865905 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.866367 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.866375 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.866376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.866449 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.866733 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.866738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.866739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.867002 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:42.867112 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:42.867394 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 04:37:42.867513 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:37:42.868902 2894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:37:42.871063 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.859720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40924 (local address 127.15.228.126:39663)
0808 04:37:42.859871 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:42.859875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:42.859897 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:37:42.860045 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:42.860049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:42.860142 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:42.860231 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:42.861111 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.861686 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.862738 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.863036 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.863154 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.866329 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:42.866333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:42.866336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:42.866370 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:42.868507 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.869111 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.869117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.869118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.869210 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.869704 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.869707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.869709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.870092 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:42.870298 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:42.870563 (+ 265us) server_negotiation.cc:299] Negotiation successful
0808 04:37:42.870699 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 04:37:42 dist-test-slave-drl0 krb5kdc[2560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163862, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.124@KRBTEST.COM for kudu/127.15.228.124@KRBTEST.COM
Aug 08 04:37:42 dist-test-slave-drl0 krb5kdc[2560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.125@KRBTEST.COM
I20260808 04:37:42.880802 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.869609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:46219 (local address 127.0.0.1:47000)
0808 04:37:42.870391 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.870404 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:42.870541 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:42.871165 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:42.871169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:42.871457 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:42.871645 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:42.871654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.873005 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.873008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:42.873682 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:42.873691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.874191 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.874195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:42.874280 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.874759 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:42.874771 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:42.876094 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:42.878840 (+ 2746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:42.878844 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:42.878847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:42.879174 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:42.879586 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:42.879589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:42.879592 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:42.879693 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:42.880285 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:42.880289 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:42.880400 (+ 111us) client_negotiation.cc:769] Sending connection context
0808 04:37:42.880523 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 04:37:42.880655 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711}
I20260808 04:37:42.881191 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.869699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.15.228.124:46219)
0808 04:37:42.869841 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:42.869845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:42.870489 (+ 644us) server_negotiation.cc:407] Connection header received
0808 04:37:42.870741 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:42.870756 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:42.870877 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:42.871105 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:42.871872 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.872821 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.873815 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.874073 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.874193 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.876268 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:42.876271 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:42.876274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:42.876298 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:42.878659 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.879356 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.879361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.879364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.879469 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.879874 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.879878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.879880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.880147 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:42.880262 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:42.880759 (+ 497us) server_negotiation.cc:299] Negotiation successful
0808 04:37:42.880965 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:37:42.884379 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.873370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56688 (local address 127.15.228.125:46545)
0808 04:37:42.873504 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:42.873507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:42.873519 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:37:42.873548 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:42.873550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:42.873585 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:42.873673 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:42.874307 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.875024 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.875939 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.876294 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.876380 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.879918 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:42.879924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:42.879930 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:42.879961 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:42.882107 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.882727 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.882732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.882734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.882853 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.883203 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.883211 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.883216 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.883502 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:42.883765 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:42.883919 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 04:37:42.884066 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 04:37:42.886277 2894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc: Bootstrap starting.
I20260808 04:37:42.889205 2894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc: Neither blocks nor log segments found. Creating new log.
I20260808 04:37:42.890348 2894 log.cc:824] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc: Log is configured to *not* fsync() on all Append() calls
I20260808 04:37:42.890432 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:37:42.892621 2894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc: No bootstrap required, opened a new log
I20260808 04:37:42.897993 2894 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5605e07666bd45f3877473ed2a562cef" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } } peers { permanent_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } } peers { permanent_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } }
I20260808 04:37:42.898437 2894 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:37:42.898542 2894 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2580cee4b2f4f11a4ce89730cab66dc, State: Initialized, Role: FOLLOWER
I20260808 04:37:42.899482 2894 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5605e07666bd45f3877473ed2a562cef" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } } peers { permanent_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } } peers { permanent_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } }
I20260808 04:37:42.900440 2914 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5605e07666bd45f3877473ed2a562cef" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } } peers { permanent_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } } peers { permanent_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } } }
I20260808 04:37:42.901522 2914 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:37:42.902119 2894 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:37:42.904913 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.893824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.15.228.125:46545)
0808 04:37:42.893968 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:42.893973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:42.894339 (+ 366us) server_negotiation.cc:407] Connection header received
0808 04:37:42.894609 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:42.894615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:42.894675 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:42.894778 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:42.895751 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.896738 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.897910 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.898284 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.898425 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.900553 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:42.900557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:42.900559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:42.900612 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:42.902849 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.903453 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.903458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.903460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.903577 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.903892 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.903895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.903897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.904110 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:42.904212 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:42.904555 (+ 343us) server_negotiation.cc:299] Negotiation successful
0808 04:37:42.904692 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:37:42.905421 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.893685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40936 (local address 127.15.228.126:39663)
0808 04:37:42.894072 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:42.894076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:42.894502 (+ 426us) server_negotiation.cc:407] Connection header received
0808 04:37:42.894773 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:42.894784 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:42.894861 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:42.895050 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:42.895784 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.896304 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.897269 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.897611 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.897685 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.900210 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:42.900214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:42.900216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:42.900260 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:42.902052 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.902555 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.902559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.902561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.902653 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.903268 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.903274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.903276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.903665 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:42.903858 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:42.905037 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0808 04:37:42.905219 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307}
W20260808 04:37:42.910470 2926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:37:42.910669 2926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:37:42.908672 2799 master_runner.cc:186] Error getting master registration for 127.15.228.126:39663: OK, instance_id { permanent_uuid: "5605e07666bd45f3877473ed2a562cef" instance_seqno: 1786163862307452 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:37:42.913887 2799 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:37:42.916626 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.894130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.15.228.124:46219)
0808 04:37:42.894271 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:42.894276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:42.904842 (+ 10566us) server_negotiation.cc:407] Connection header received
0808 04:37:42.906329 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:42.906333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:42.906387 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:42.906443 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:42.907481 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.908059 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.909319 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.909646 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.909761 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.912067 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:42.912072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:42.912075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:42.912115 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:42.914305 (+ 2190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.914843 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.914849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.914851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.914994 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.915486 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.915490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.915491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.915810 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:42.915964 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:42.916290 (+ 326us) server_negotiation.cc:299] Negotiation successful
0808 04:37:42.916431 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:37:42.959764 2792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } attempt: 4
I20260808 04:37:42.962450 2685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } attempt: 4
Aug 08 04:37:42 dist-test-slave-drl0 krb5kdc[2560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163862, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.125@KRBTEST.COM for kudu/127.15.228.124@KRBTEST.COM
Aug 08 04:37:42 dist-test-slave-drl0 krb5kdc[2560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163862, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for kudu/127.15.228.124@KRBTEST.COM
I20260808 04:37:42.970695 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.960326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:46219 (local address 127.0.0.1:47026)
0808 04:37:42.960496 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.960517 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:42.960615 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:42.961204 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:42.961207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:42.961434 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:42.961534 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:42.961540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.962377 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.962381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:42.963231 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:42.963247 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.963788 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.963792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:42.963889 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.964678 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:42.964694 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:42.966469 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:42.968798 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:42.968802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:42.968805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:42.969116 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:42.969452 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:42.969459 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:42.969464 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:42.969588 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:42.970089 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:42.970093 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:42.970216 (+ 123us) client_negotiation.cc:769] Sending connection context
0808 04:37:42.970343 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 04:37:42.970496 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 04:37:42.971001 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.960781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47026 (local address 127.15.228.124:46219)
0808 04:37:42.960961 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:42.960967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:42.960981 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:37:42.961022 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:42.961025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:42.961075 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:42.961167 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:42.961662 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.962245 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.963373 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.963665 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.963796 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.966641 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:42.966645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:42.966649 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:42.966677 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:42.968668 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.969240 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.969244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.969247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.969347 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.969715 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.969721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.969723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.969960 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:42.970082 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:42.970442 (+ 360us) server_negotiation.cc:299] Negotiation successful
0808 04:37:42.970577 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 04:37:42.973248 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.962893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:46219 (local address 127.0.0.1:47036)
0808 04:37:42.963093 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:42.963107 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:42.963180 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:42.963670 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:42.963673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:42.963918 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:42.964027 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:42.964033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.965393 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.965397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:42.965978 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:42.965987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.966552 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.966556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:42.966653 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.967247 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:42.967269 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:42.968808 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:42.971164 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:42.971169 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:42.971171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:42.971452 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:42.971825 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:42.971829 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:42.971831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:42.971936 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:42.972661 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:42.972665 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:42.972783 (+ 118us) client_negotiation.cc:769] Sending connection context
0808 04:37:42.972912 (+ 129us) client_negotiation.cc:240] Negotiation successful
0808 04:37:42.973063 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260808 04:37:42.974180 2929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:42.963053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47036 (local address 127.15.228.124:46219)
0808 04:37:42.963429 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:42.963442 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:42.963455 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:37:42.963489 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:42.963492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:42.963531 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:42.963616 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:42.964236 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.965276 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.966120 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:42.966422 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:42.966548 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:42.968997 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:42.969003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:42.969007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:42.969038 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:42.971009 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.971595 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.971601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.971606 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.971702 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:42.972063 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:42.972068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:42.972073 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:42.972538 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:42.972650 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:42.973836 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0808 04:37:42.973978 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":155,"threads_started":1}
I20260808 04:37:42.976318 2792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2: Bootstrap starting.
I20260808 04:37:42.977550 2685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef: Bootstrap starting.
I20260808 04:37:42.978981 2792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2: Neither blocks nor log segments found. Creating new log.
I20260808 04:37:42.979827 2792 log.cc:824] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2: Log is configured to *not* fsync() on all Append() calls
I20260808 04:37:42.979988 2685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef: Neither blocks nor log segments found. Creating new log.
I20260808 04:37:42.980810 2685 log.cc:824] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef: Log is configured to *not* fsync() on all Append() calls
I20260808 04:37:42.982568 2792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2: No bootstrap required, opened a new log
I20260808 04:37:42.982923 2685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef: No bootstrap required, opened a new log
I20260808 04:37:42.987062 2685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5605e07666bd45f3877473ed2a562cef" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } } peers { permanent_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } } peers { permanent_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } }
I20260808 04:37:42.987326 2685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:37:42.987375 2685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5605e07666bd45f3877473ed2a562cef, State: Initialized, Role: FOLLOWER
I20260808 04:37:42.987918 2685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5605e07666bd45f3877473ed2a562cef" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } } peers { permanent_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } } peers { permanent_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } }
I20260808 04:37:42.988436 2792 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5605e07666bd45f3877473ed2a562cef" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } } peers { permanent_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } } peers { permanent_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } }
I20260808 04:37:42.988664 2933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5605e07666bd45f3877473ed2a562cef" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } } peers { permanent_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } } peers { permanent_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } } }
I20260808 04:37:42.988910 2933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:37:42.988925 2792 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:37:42.989025 2792 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad79d179d9604f34afe73e6aca8f0fd2, State: Initialized, Role: FOLLOWER
I20260808 04:37:42.989411 2685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:37:42.989701 2792 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5605e07666bd45f3877473ed2a562cef" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } } peers { permanent_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } } peers { permanent_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } }
I20260808 04:37:42.990375 2935 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5605e07666bd45f3877473ed2a562cef" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } } peers { permanent_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } } peers { permanent_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } } }
I20260808 04:37:42.990697 2935 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:37:42.991854 2792 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 04:37:42.995543 2949 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:37:42.997081 2949 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260808 04:37:42.998214 2969 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:37:42.998322 2969 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:37:43.002499 2693 master_runner.cc:263] No leader master found from master ad79d179d9604f34afe73e6aca8f0fd2
I20260808 04:37:43.002590 2693 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:37:43.002745 2585 master_runner.cc:263] No leader master found from master 5605e07666bd45f3877473ed2a562cef
I20260808 04:37:43.003114 2585 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 04:37:43.044974 2723 tablet.cc:2369] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:37:43.052630 2614 tablet.cc:2369] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:37:43.068928 2828 tablet.cc:2369] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:37:43.106356 2935 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:37:43.106779 2935 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5605e07666bd45f3877473ed2a562cef" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } } peers { permanent_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } } peers { permanent_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } }
I20260808 04:37:43.108135 2935 leader_election.cc:288] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5605e07666bd45f3877473ed2a562cef (127.15.228.126:39663), f2580cee4b2f4f11a4ce89730cab66dc (127.15.228.124:46219)
I20260808 04:37:43.109306 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5605e07666bd45f3877473ed2a562cef" is_pre_election: true
I20260808 04:37:43.109661 2643 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad79d179d9604f34afe73e6aca8f0fd2 in term 0.
I20260808 04:37:43.110334 2717 leader_election.cc:302] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5605e07666bd45f3877473ed2a562cef, ad79d179d9604f34afe73e6aca8f0fd2; no voters:
I20260808 04:37:43.111186 2854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" is_pre_election: true
I20260808 04:37:43.111500 2854 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad79d179d9604f34afe73e6aca8f0fd2 in term 0.
I20260808 04:37:43.112210 2935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:37:43.112314 2935 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:37:43.112363 2935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:37:43.115326 2935 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5605e07666bd45f3877473ed2a562cef" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } } peers { permanent_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } } peers { permanent_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } }
I20260808 04:37:43.116811 2643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5605e07666bd45f3877473ed2a562cef"
I20260808 04:37:43.117035 2643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:37:43.119232 2643 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad79d179d9604f34afe73e6aca8f0fd2 in term 1.
I20260808 04:37:43.119966 2717 leader_election.cc:302] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5605e07666bd45f3877473ed2a562cef, ad79d179d9604f34afe73e6aca8f0fd2; no voters:
I20260808 04:37:43.125188 2854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2580cee4b2f4f11a4ce89730cab66dc"
I20260808 04:37:43.125387 2854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:37:43.126886 2854 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad79d179d9604f34afe73e6aca8f0fd2 in term 1.
I20260808 04:37:43.127528 2935 leader_election.cc:288] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [CANDIDATE]: Term 1 election: Requested vote from peers 5605e07666bd45f3877473ed2a562cef (127.15.228.126:39663), f2580cee4b2f4f11a4ce89730cab66dc (127.15.228.124:46219)
I20260808 04:37:43.127813 2935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:37:43.127962 2935 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [term 1 LEADER]: Becoming Leader. State: Replica: ad79d179d9604f34afe73e6aca8f0fd2, State: Running, Role: LEADER
I20260808 04:37:43.128469 2935 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5605e07666bd45f3877473ed2a562cef" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } } peers { permanent_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } } peers { permanent_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } }
I20260808 04:37:43.130813 2935 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad79d179d9604f34afe73e6aca8f0fd2. Latest consensus state: current_term: 1 leader_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5605e07666bd45f3877473ed2a562cef" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } } peers { permanent_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } } peers { permanent_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } } }
I20260808 04:37:43.131031 2935 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [sys.catalog]: This master's current role is: LEADER
I20260808 04:37:43.131570 2974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:37:43.132781 2974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:37:43.144994 2854 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc [term 1 FOLLOWER]: Refusing update from remote peer ad79d179d9604f34afe73e6aca8f0fd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:37:43.145974 2643 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef [term 1 FOLLOWER]: Refusing update from remote peer ad79d179d9604f34afe73e6aca8f0fd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:37:43.146428 2972 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:37:43.146962 2972 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5605e07666bd45f3877473ed2a562cef" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:37:43.151276 2933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad79d179d9604f34afe73e6aca8f0fd2. Latest consensus state: current_term: 1 leader_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5605e07666bd45f3877473ed2a562cef" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } } peers { permanent_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } } peers { permanent_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } } }
I20260808 04:37:43.151453 2933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:37:43.151894 2914 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad79d179d9604f34afe73e6aca8f0fd2. Latest consensus state: current_term: 1 leader_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5605e07666bd45f3877473ed2a562cef" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } } peers { permanent_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } } peers { permanent_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } } }
I20260808 04:37:43.152045 2914 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:37:43.154767 2972 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5605e07666bd45f3877473ed2a562cef" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } } peers { permanent_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } } peers { permanent_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } } }
I20260808 04:37:43.154945 2972 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [sys.catalog]: This master's current role is: LEADER
I20260808 04:37:43.155396 2933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5605e07666bd45f3877473ed2a562cef" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } } peers { permanent_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } } peers { permanent_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } } }
I20260808 04:37:43.155524 2933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:37:43.155867 2977 mvcc.cc:205] Tried to move back new op lower bound from 7316127183427461120 to 7316127183378354176. Current Snapshot: MvccSnapshot[applied={T|T < 7316127183427461120}]
I20260808 04:37:43.157941 2978 mvcc.cc:205] Tried to move back new op lower bound from 7316127183427461120 to 7316127183378354176. Current Snapshot: MvccSnapshot[applied={T|T < 7316127183427461120}]
I20260808 04:37:43.160686 2974 catalog_manager.cc:1347] Generated new cluster ID: 9cab1cf1a0d745029f9211869e1cb12a
I20260808 04:37:43.160758 2974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:37:43.166623 2935 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5605e07666bd45f3877473ed2a562cef" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } } peers { permanent_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } } peers { permanent_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } } }
I20260808 04:37:43.166764 2935 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2 [sys.catalog]: This master's current role is: LEADER
I20260808 04:37:43.169668 2914 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5605e07666bd45f3877473ed2a562cef" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 39663 } } peers { permanent_uuid: "ad79d179d9604f34afe73e6aca8f0fd2" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 46545 } } peers { permanent_uuid: "f2580cee4b2f4f11a4ce89730cab66dc" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 46219 } } }
I20260808 04:37:43.169809 2914 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:37:43.180017 2974 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:37:43.181854 2974 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:37:43.201090 2974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad79d179d9604f34afe73e6aca8f0fd2: Generated new TSK 0
I20260808 04:37:43.202064 2974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:37:43.912189 2926 catalog_manager.cc:1259] Loaded cluster ID: 9cab1cf1a0d745029f9211869e1cb12a
I20260808 04:37:43.912292 2926 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc: loading cluster ID for follower catalog manager: success
I20260808 04:37:43.914740 2926 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc: acquiring CA information for follower catalog manager: success
I20260808 04:37:43.916505 2926 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f2580cee4b2f4f11a4ce89730cab66dc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:37:43.998780 2949 catalog_manager.cc:1259] Loaded cluster ID: 9cab1cf1a0d745029f9211869e1cb12a
I20260808 04:37:43.998883 2949 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef: loading cluster ID for follower catalog manager: success
I20260808 04:37:44.000903 2949 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef: acquiring CA information for follower catalog manager: success
I20260808 04:37:44.002229 2949 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5605e07666bd45f3877473ed2a562cef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:37:44.281350 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 2585
I20260808 04:37:44.300503 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 2693
I20260808 04:37:44.315106 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 2799
2026-08-08T04:37:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3704 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:37:44 dist-test-slave-drl0 krb5kdc[2997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:37:44 dist-test-slave-drl0 krb5kdc[2997](info): set up 2 sockets
Aug 08 04:37:44 dist-test-slave-drl0 krb5kdc[2997](info): commencing operation
krb5kdc: starting...
W20260808 04:37:45.623389 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.281s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:37:45 dist-test-slave-drl0 krb5kdc[2997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:37:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:37:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:37:45.775379 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:41651
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:39111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:41651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:37:45.903355 3020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:45.903641 3020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:45.903684 3020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:45.907982 3020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:37:45.908056 3020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:45.908077 3020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:45.908093 3020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:37:45.908109 3020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:37:45.913576 3020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:39111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:41651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:41651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:45.914674 3020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:45.915760 3020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:45.926173 3025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:45.926363 3029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:45.926573 3026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:45.926983 3020 server_base.cc:1034] running on GCE node
I20260808 04:37:45.927582 3020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:45.928821 3020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:45.930084 3020 hybrid_clock.cc:648] HybridClock initialized: now 1786163865930048 us; error 61 us; skew 500 ppm
Aug 08 04:37:45 dist-test-slave-drl0 krb5kdc[2997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163865, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:45.935189 3020 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:37:45.936738 3020 webserver.cc:460] Webserver started at http://127.15.228.126:38061/ using document root <none> and password file <none>
I20260808 04:37:45.937583 3020 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:45.937700 3020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:37:45.938203 3020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:37:45.942112 3020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "ee1a4d414b5742d8b3790119cc0013ba"
format_stamp: "Formatted at 2026-08-08 04:37:45 on dist-test-slave-drl0"
server_key: "a4765b74fa2079a06440af3fc3e51a06"
server_key_iv: "97fba7e14b263b9a418d974325dd561d"
server_key_version: "encryptionkey@0"
I20260808 04:37:45.943046 3020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "ee1a4d414b5742d8b3790119cc0013ba"
format_stamp: "Formatted at 2026-08-08 04:37:45 on dist-test-slave-drl0"
server_key: "a4765b74fa2079a06440af3fc3e51a06"
server_key_iv: "97fba7e14b263b9a418d974325dd561d"
server_key_version: "encryptionkey@0"
I20260808 04:37:45.948580 3020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260808 04:37:45.951781 3038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:45.953579 3020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:37:45.953763 3020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "ee1a4d414b5742d8b3790119cc0013ba"
format_stamp: "Formatted at 2026-08-08 04:37:45 on dist-test-slave-drl0"
server_key: "a4765b74fa2079a06440af3fc3e51a06"
server_key_iv: "97fba7e14b263b9a418d974325dd561d"
server_key_version: "encryptionkey@0"
I20260808 04:37:45.953908 3020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:45.982295 3020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:45.985764 3020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:45.986016 3020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:45.994271 3020 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:41651
I20260808 04:37:45.994300 3115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:41651 every 8 connection(s)
I20260808 04:37:45.995492 3020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:37:45.998481 3117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:37:46.004343 3117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba: Bootstrap starting.
I20260808 04:37:46.005285 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 3020
I20260808 04:37:46.005488 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:37:46.006018 16273 external_mini_cluster.cc:1428] Setting key 8e5c715ed00a538a4e6a8515e9cf302c
I20260808 04:37:46.007666 3117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba: Neither blocks nor log segments found. Creating new log.
I20260808 04:37:46.008594 3117 log.cc:824] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba: Log is configured to *not* fsync() on all Append() calls
I20260808 04:37:46.010905 3117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba: No bootstrap required, opened a new log
I20260808 04:37:46.015131 3117 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1a4d414b5742d8b3790119cc0013ba" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41651 } }
I20260808 04:37:46.015350 3117 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:37:46.015432 3117 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee1a4d414b5742d8b3790119cc0013ba, State: Initialized, Role: FOLLOWER
I20260808 04:37:46.015865 3117 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1a4d414b5742d8b3790119cc0013ba" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41651 } }
I20260808 04:37:46.016017 3117 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:37:46.016088 3117 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:37:46.016201 3117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba [term 0 FOLLOWER]: Advancing to term 1
Aug 08 04:37:46 dist-test-slave-drl0 krb5kdc[2997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:37:46.017371 3117 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1a4d414b5742d8b3790119cc0013ba" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41651 } }
I20260808 04:37:46.017774 3117 leader_election.cc:302] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee1a4d414b5742d8b3790119cc0013ba; no voters:
I20260808 04:37:46.018199 3117 leader_election.cc:288] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:37:46.018472 3124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:37:46.018863 3124 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba [term 1 LEADER]: Becoming Leader. State: Replica: ee1a4d414b5742d8b3790119cc0013ba, State: Running, Role: LEADER
I20260808 04:37:46.019207 3124 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1a4d414b5742d8b3790119cc0013ba" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41651 } }
I20260808 04:37:46.019685 3117 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:37:46.020936 3125 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee1a4d414b5742d8b3790119cc0013ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1a4d414b5742d8b3790119cc0013ba" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41651 } } }
I20260808 04:37:46.021081 3125 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba [sys.catalog]: This master's current role is: LEADER
I20260808 04:37:46.021406 3127 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee1a4d414b5742d8b3790119cc0013ba. Latest consensus state: current_term: 1 leader_uuid: "ee1a4d414b5742d8b3790119cc0013ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1a4d414b5742d8b3790119cc0013ba" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41651 } } }
I20260808 04:37:46.021502 3127 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba [sys.catalog]: This master's current role is: LEADER
I20260808 04:37:46.023886 3120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:46.007275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.15.228.126:41651)
0808 04:37:46.007840 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:46.007877 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:46.007928 (+ 51us) server_negotiation.cc:407] Connection header received
0808 04:37:46.008817 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:46.008840 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:46.009226 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:46.009580 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:46.010322 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:46.011841 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:46.013064 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:46.014016 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:46.014208 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:46.017373 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:46.017383 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:46.017403 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:46.017438 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:46.019802 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:46.020395 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:46.020401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:46.020408 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:46.020559 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:46.020968 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:46.020973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:46.020975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:46.021557 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:46.021769 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:46.022611 (+ 842us) server_negotiation.cc:299] Negotiation successful
0808 04:37:46.022914 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":163,"threads_started":1}
W20260808 04:37:46.025158 3144 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:37:46.025234 3144 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:37:46.025293 3138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:37:46.026151 3138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:37:46.027122 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:37:46.031746 3138 catalog_manager.cc:1347] Generated new cluster ID: c693b8e783d749c08f207df1ccc0a0a6
I20260808 04:37:46.031816 3138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:37:46.053537 3138 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:37:46.055400 3138 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:37:46.062795 3138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee1a4d414b5742d8b3790119cc0013ba: Generated new TSK 0
I20260808 04:37:46.063984 3138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:37:46.084919 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 3020
2026-08-08T04:37:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1776 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:37:46 dist-test-slave-drl0 krb5kdc[3152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:37:46 dist-test-slave-drl0 krb5kdc[3152](info): set up 2 sockets
Aug 08 04:37:46 dist-test-slave-drl0 krb5kdc[3152](info): commencing operation
krb5kdc: starting...
W20260808 04:37:47.374260 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.255s user 0.001s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:37:47 dist-test-slave-drl0 krb5kdc[3152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:37:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:37:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:37:47.523070 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:34873
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:34873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:37:47.660447 3169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:47.660748 3169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:47.660791 3169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:47.665033 3169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:37:47.665096 3169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:47.665131 3169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:47.665150 3169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:37:47.665165 3169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:37:47.671908 3169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:34873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:34873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:47.673226 3169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:47.674340 3169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:47.683245 3175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:47.683286 3176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:47.683318 3179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:47.683599 3169 server_base.cc:1034] running on GCE node
I20260808 04:37:47.684204 3169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:47.685329 3169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:47.686563 3169 hybrid_clock.cc:648] HybridClock initialized: now 1786163867686496 us; error 75 us; skew 500 ppm
Aug 08 04:37:47 dist-test-slave-drl0 krb5kdc[3152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163867, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:47.693446 3169 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:37:47.694407 3169 webserver.cc:460] Webserver started at http://127.15.228.126:32915/ using document root <none> and password file <none>
I20260808 04:37:47.695070 3169 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:47.695168 3169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:37:47.695425 3169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:37:47.697587 3169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "53d25bfe01ec48abb6226f2b1aef9afa"
format_stamp: "Formatted at 2026-08-08 04:37:47 on dist-test-slave-drl0"
server_key: "bf07bf9316c10744bf43fb6ee7a8cd26"
server_key_iv: "40fa369efbf0a463793da8f15502ba74"
server_key_version: "encryptionkey@0"
I20260808 04:37:47.698203 3169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "53d25bfe01ec48abb6226f2b1aef9afa"
format_stamp: "Formatted at 2026-08-08 04:37:47 on dist-test-slave-drl0"
server_key: "bf07bf9316c10744bf43fb6ee7a8cd26"
server_key_iv: "40fa369efbf0a463793da8f15502ba74"
server_key_version: "encryptionkey@0"
I20260808 04:37:47.702116 3169 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:37:47.704810 3190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:47.706419 3169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:37:47.706584 3169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "53d25bfe01ec48abb6226f2b1aef9afa"
format_stamp: "Formatted at 2026-08-08 04:37:47 on dist-test-slave-drl0"
server_key: "bf07bf9316c10744bf43fb6ee7a8cd26"
server_key_iv: "40fa369efbf0a463793da8f15502ba74"
server_key_version: "encryptionkey@0"
I20260808 04:37:47.706708 3169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:47.725195 3169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:47.728611 3169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:47.728840 3169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:47.737365 3169 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:34873
I20260808 04:37:47.737674 3262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:34873 every 8 connection(s)
I20260808 04:37:47.738973 3169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:37:47.742540 3263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:37:47.743206 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 3169
I20260808 04:37:47.743368 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:37:47.743675 16273 external_mini_cluster.cc:1428] Setting key 952d95b93ceb2d6e9569d144cd82e70c
I20260808 04:37:47.749742 3263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa: Bootstrap starting.
I20260808 04:37:47.752825 3263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa: Neither blocks nor log segments found. Creating new log.
I20260808 04:37:47.753877 3263 log.cc:824] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:37:47 dist-test-slave-drl0 krb5kdc[3152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:37:47.756361 3263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa: No bootstrap required, opened a new log
I20260808 04:37:47.760278 3263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34873 } }
I20260808 04:37:47.760568 3263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:37:47.760667 3263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d25bfe01ec48abb6226f2b1aef9afa, State: Initialized, Role: FOLLOWER
I20260808 04:37:47.761310 3263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34873 } }
I20260808 04:37:47.761492 3263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:37:47.761607 3263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:37:47.761711 3263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:37:47.762521 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:47.745096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38198 (local address 127.15.228.126:34873)
0808 04:37:47.745561 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:47.745577 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:47.745629 (+ 52us) server_negotiation.cc:407] Connection header received
0808 04:37:47.746743 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:47.746769 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:47.747446 (+ 677us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:47.747740 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:47.748817 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:47.750391 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:47.751782 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:47.752288 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:47.752423 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:47.755802 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:47.755807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:47.755823 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:47.755852 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:47.757673 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:47.758159 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:47.758165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:47.758170 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:47.758285 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:47.758832 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:47.758839 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:47.758841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:47.759803 (+ 962us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:47.760261 (+ 458us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:47.760669 (+ 408us) server_negotiation.cc:299] Negotiation successful
0808 04:37:47.761191 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":166,"threads_started":1}
I20260808 04:37:47.762825 3263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34873 } }
I20260808 04:37:47.763471 3263 leader_election.cc:302] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53d25bfe01ec48abb6226f2b1aef9afa; no voters:
I20260808 04:37:47.764008 3263 leader_election.cc:288] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:37:47.764315 3270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:37:47.764680 3270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 1 LEADER]: Becoming Leader. State: Replica: 53d25bfe01ec48abb6226f2b1aef9afa, State: Running, Role: LEADER
I20260808 04:37:47.765067 3270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34873 } }
I20260808 04:37:47.765877 3263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:37:47.767045 3272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34873 } } }
I20260808 04:37:47.767186 3272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [sys.catalog]: This master's current role is: LEADER
I20260808 04:37:47.767590 3283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:37:47.767534 3273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53d25bfe01ec48abb6226f2b1aef9afa. Latest consensus state: current_term: 1 leader_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34873 } } }
I20260808 04:37:47.768246 3273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [sys.catalog]: This master's current role is: LEADER
I20260808 04:37:47.770545 3283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:37:47.776352 3283 catalog_manager.cc:1347] Generated new cluster ID: 273838deed8842cdb8d537d1ee70de99
I20260808 04:37:47.776444 3283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:37:47.783473 3283 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:37:47.784379 3283 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:37:47.790576 3283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa: Generated new TSK 0
I20260808 04:37:47.791430 3283 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
I20260808 04:37:47.831593 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:34873
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:37:47.957664 3297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:47.957993 3297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:47.958042 3297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:47.962474 3297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:47.962549 3297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:47.962608 3297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:37:47.968784 3297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:34873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:47.969969 3297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:47.971077 3297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:47.980809 3302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:47.980886 3306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:47.981022 3297 server_base.cc:1034] running on GCE node
W20260808 04:37:47.981084 3304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:47.981578 3297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:47.982142 3297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:47.983343 3297 hybrid_clock.cc:648] HybridClock initialized: now 1786163867983287 us; error 65 us; skew 500 ppm
Aug 08 04:37:47 dist-test-slave-drl0 krb5kdc[3152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163867, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:47.986495 3297 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:37:47.987566 3297 webserver.cc:460] Webserver started at http://127.15.228.65:42241/ using document root <none> and password file <none>
I20260808 04:37:47.988257 3297 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:47.988333 3297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:37:47.988559 3297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:37:47.990702 3297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "3e1fc50de4bb4098b2b77d3976fbcf03"
format_stamp: "Formatted at 2026-08-08 04:37:47 on dist-test-slave-drl0"
server_key: "34b67d3a73530ee9e7342b88cc1968e0"
server_key_iv: "43f533ec3d898e5673d0ce952e642c74"
server_key_version: "encryptionkey@0"
I20260808 04:37:47.991376 3297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "3e1fc50de4bb4098b2b77d3976fbcf03"
format_stamp: "Formatted at 2026-08-08 04:37:47 on dist-test-slave-drl0"
server_key: "34b67d3a73530ee9e7342b88cc1968e0"
server_key_iv: "43f533ec3d898e5673d0ce952e642c74"
server_key_version: "encryptionkey@0"
I20260808 04:37:47.996454 3297 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260808 04:37:48.000105 3314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:48.001974 3297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260808 04:37:48.002238 3297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "3e1fc50de4bb4098b2b77d3976fbcf03"
format_stamp: "Formatted at 2026-08-08 04:37:47 on dist-test-slave-drl0"
server_key: "34b67d3a73530ee9e7342b88cc1968e0"
server_key_iv: "43f533ec3d898e5673d0ce952e642c74"
server_key_version: "encryptionkey@0"
I20260808 04:37:48.002419 3297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:48.015492 3297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:48.019264 3297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:48.019487 3297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:48.020138 3297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:37:48.021042 3297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:37:48.021116 3297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:48.021184 3297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:37:48.021235 3297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:48.032083 3297 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:40511
I20260808 04:37:48.032155 3472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:40511 every 8 connection(s)
I20260808 04:37:48.033056 3297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:37:48.042042 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 3297
I20260808 04:37:48.042220 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:37:48.042578 16273 external_mini_cluster.cc:1428] Setting key 1e9c5710597924c3cd1e01a2e63342ca
Aug 08 04:37:48 dist-test-slave-drl0 krb5kdc[3152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163867, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:37:48.049638 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:48.035153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:48923 (local address 127.15.228.126:34873)
0808 04:37:48.035423 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:48.035429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:48.036283 (+ 854us) server_negotiation.cc:407] Connection header received
0808 04:37:48.037839 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:48.037842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:48.037898 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:48.037994 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:48.039352 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.040062 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.041299 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.041551 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.041643 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:48.044587 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:48.044591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:48.044594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:48.044620 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:48.046200 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:48.046915 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:48.046918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:48.046920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:48.047089 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:48.047941 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:48.047947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:48.047950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:48.048234 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:48.048358 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:48.049214 (+ 856us) server_negotiation.cc:299] Negotiation successful
0808 04:37:48.049375 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260808 04:37:48.051103 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:48.035541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:48923)
0808 04:37:48.036070 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:48.036115 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:48.037543 (+ 1428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:48.038174 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:48.038180 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:48.038573 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:48.039106 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:48.039120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.040243 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.040247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:48.041165 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:48.041175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.041761 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.041769 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:48.042014 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:48.042803 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:48.042826 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:48.044442 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:48.046341 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:48.046350 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:48.046370 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:48.046764 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:48.047194 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:48.047200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:48.047203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:48.047696 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:48.048420 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:48.048429 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:48.048852 (+ 423us) client_negotiation.cc:769] Sending connection context
0808 04:37:48.049088 (+ 236us) client_negotiation.cc:240] Negotiation successful
0808 04:37:48.049581 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":129,"threads_started":1}
I20260808 04:37:48.052989 3473 heartbeater.cc:347] Connected to a master server at 127.15.228.126:34873
I20260808 04:37:48.053277 3473 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:48.053848 3473 heartbeater.cc:511] Master 127.15.228.126:34873 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.228.66@KRBTEST.COM; defaulting to no policy
I20260808 04:37:48.055991 3212 ts_manager.cc:194] Registered new tserver with Master: 3e1fc50de4bb4098b2b77d3976fbcf03 (127.15.228.65:40511)
I20260808 04:37:48.057399 3212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:48923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
I20260808 04:37:48.079294 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:34873
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:37:48.205137 3481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:48.205411 3481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:48.205457 3481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:48.209663 3481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:48.209731 3481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:48.209791 3481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:37:48.215790 3481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:34873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:48.216940 3481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:48.218046 3481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:48.228174 3492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:48.228197 3487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:48.228297 3490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:48.228703 3481 server_base.cc:1034] running on GCE node
I20260808 04:37:48.229447 3481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:48.230475 3481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:48.231688 3481 hybrid_clock.cc:648] HybridClock initialized: now 1786163868231667 us; error 36 us; skew 500 ppm
Aug 08 04:37:48 dist-test-slave-drl0 krb5kdc[3152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163868, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:48.235765 3481 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:37:48.236917 3481 webserver.cc:460] Webserver started at http://127.15.228.66:42265/ using document root <none> and password file <none>
I20260808 04:37:48.237511 3481 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:48.237607 3481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:37:48.237843 3481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:37:48.240463 3481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "c5fcc7345b6c4ccaa21f38c606ada79a"
format_stamp: "Formatted at 2026-08-08 04:37:48 on dist-test-slave-drl0"
server_key: "c8f9044eb78ee1829f7cd8d933e611d8"
server_key_iv: "eb154396f3a905ba580482b27ec47501"
server_key_version: "encryptionkey@0"
I20260808 04:37:48.241108 3481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "c5fcc7345b6c4ccaa21f38c606ada79a"
format_stamp: "Formatted at 2026-08-08 04:37:48 on dist-test-slave-drl0"
server_key: "c8f9044eb78ee1829f7cd8d933e611d8"
server_key_iv: "eb154396f3a905ba580482b27ec47501"
server_key_version: "encryptionkey@0"
I20260808 04:37:48.245105 3481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:37:48.247944 3508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:48.249965 3481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:37:48.250221 3481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "c5fcc7345b6c4ccaa21f38c606ada79a"
format_stamp: "Formatted at 2026-08-08 04:37:48 on dist-test-slave-drl0"
server_key: "c8f9044eb78ee1829f7cd8d933e611d8"
server_key_iv: "eb154396f3a905ba580482b27ec47501"
server_key_version: "encryptionkey@0"
I20260808 04:37:48.250365 3481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:48.267012 3481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:48.271059 3481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:48.271261 3481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:48.271929 3481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:37:48.272802 3481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:37:48.272855 3481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:48.272897 3481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:37:48.272948 3481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:48.283177 3481 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:33473
I20260808 04:37:48.283281 3666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:33473 every 8 connection(s)
I20260808 04:37:48.284201 3481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:37:48.289356 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 3481
I20260808 04:37:48.289520 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:37:48.289791 16273 external_mini_cluster.cc:1428] Setting key e2d32e649da4cba8b556f2f319cc3bf2
Aug 08 04:37:48 dist-test-slave-drl0 krb5kdc[3152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163868, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.228.67@KRBTEST.COM; defaulting to no policy
I20260808 04:37:48.303257 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:48.286270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:49127 (local address 127.15.228.126:34873)
0808 04:37:48.286442 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:48.286446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:48.287339 (+ 893us) server_negotiation.cc:407] Connection header received
0808 04:37:48.289822 (+ 2483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:48.289826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:48.289901 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:48.290014 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:48.292093 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.292972 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.294825 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.295167 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.295318 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:48.298654 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:48.298658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:48.298660 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:48.298691 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:48.300355 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:48.300996 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:48.301000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:48.301002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:48.301096 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:48.301493 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:48.301497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:48.301499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:48.301807 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:48.301925 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:48.302847 (+ 922us) server_negotiation.cc:299] Negotiation successful
0808 04:37:48.303064 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 04:37:48.304277 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:48.286643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.66:49127)
0808 04:37:48.287186 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:48.287238 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:48.289576 (+ 2338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:48.290197 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:48.290204 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:48.291064 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:48.291937 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:48.291955 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.293356 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.293360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:48.294603 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:48.294623 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.295401 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.295406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:48.295647 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:48.296447 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:48.296469 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:48.298400 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:48.300504 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:48.300512 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:48.300534 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:48.300857 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:48.301200 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:48.301203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:48.301205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:48.301379 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:48.301925 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:48.301934 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:48.302439 (+ 505us) client_negotiation.cc:769] Sending connection context
0808 04:37:48.302767 (+ 328us) client_negotiation.cc:240] Negotiation successful
0808 04:37:48.303202 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.67@KRBTEST.COM" created.
I20260808 04:37:48.305529 3667 heartbeater.cc:347] Connected to a master server at 127.15.228.126:34873
I20260808 04:37:48.305832 3667 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:48.306514 3667 heartbeater.cc:511] Master 127.15.228.126:34873 requested a full tablet report, sending...
I20260808 04:37:48.307971 3212 ts_manager.cc:194] Registered new tserver with Master: c5fcc7345b6c4ccaa21f38c606ada79a (127.15.228.66:33473)
I20260808 04:37:48.308916 3212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:49127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
I20260808 04:37:48.322220 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:34873
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:37:48.449395 3673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:48.449673 3673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:48.449723 3673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:48.453954 3673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:48.454020 3673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:48.454082 3673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:37:48.459553 3673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:34873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:48.460685 3673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:48.461800 3673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:48.472325 3680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:48.472325 3682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:48.472370 3679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:48.472846 3673 server_base.cc:1034] running on GCE node
I20260808 04:37:48.473553 3673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:48.474341 3673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:48.475550 3673 hybrid_clock.cc:648] HybridClock initialized: now 1786163868475523 us; error 45 us; skew 500 ppm
Aug 08 04:37:48 dist-test-slave-drl0 krb5kdc[3152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163868, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:48.478562 3673 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:37:48.479537 3673 webserver.cc:460] Webserver started at http://127.15.228.67:39225/ using document root <none> and password file <none>
I20260808 04:37:48.480135 3673 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:48.480201 3673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:37:48.480463 3673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:37:48.482599 3673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "bedaa4b0188e4673a4cefc4dca0d48ea"
format_stamp: "Formatted at 2026-08-08 04:37:48 on dist-test-slave-drl0"
server_key: "73b4dcaa4656c17d741931a8ee75af6e"
server_key_iv: "aebd2605e82b9df8ba210b22dabb2001"
server_key_version: "encryptionkey@0"
I20260808 04:37:48.483248 3673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "bedaa4b0188e4673a4cefc4dca0d48ea"
format_stamp: "Formatted at 2026-08-08 04:37:48 on dist-test-slave-drl0"
server_key: "73b4dcaa4656c17d741931a8ee75af6e"
server_key_iv: "aebd2605e82b9df8ba210b22dabb2001"
server_key_version: "encryptionkey@0"
I20260808 04:37:48.487226 3673 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:37:48.490777 3691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:48.492235 3673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:37:48.492403 3673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "bedaa4b0188e4673a4cefc4dca0d48ea"
format_stamp: "Formatted at 2026-08-08 04:37:48 on dist-test-slave-drl0"
server_key: "73b4dcaa4656c17d741931a8ee75af6e"
server_key_iv: "aebd2605e82b9df8ba210b22dabb2001"
server_key_version: "encryptionkey@0"
I20260808 04:37:48.492529 3673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:48.512619 3673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:48.516284 3673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:48.516516 3673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:48.517158 3673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:37:48.518678 3673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:37:48.518769 3673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:48.518846 3673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:37:48.518891 3673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:48.530726 3673 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:34349
I20260808 04:37:48.530751 3845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:34349 every 8 connection(s)
I20260808 04:37:48.531992 3673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:37:48.542196 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 3673
I20260808 04:37:48.542377 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:37:48.542775 16273 external_mini_cluster.cc:1428] Setting key 599ef6806c7ceb575e331b82c45f8544
Aug 08 04:37:48 dist-test-slave-drl0 krb5kdc[3152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163868, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:37:48.549618 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:48.534018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:60453 (local address 127.15.228.126:34873)
0808 04:37:48.534190 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:48.534196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:48.535494 (+ 1298us) server_negotiation.cc:407] Connection header received
0808 04:37:48.537150 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:48.537154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:48.537210 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:48.537307 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:48.538648 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.539288 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.540494 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.540710 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.540804 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:48.544758 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:48.544762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:48.544765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:48.544797 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:48.546696 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:48.547731 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:48.547737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:48.547739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:48.547844 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:48.548401 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:48.548406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:48.548408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:48.548630 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:48.548753 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:48.549314 (+ 561us) server_negotiation.cc:299] Negotiation successful
0808 04:37:48.549447 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 04:37:48.551304 3851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:48.534434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.67:60453)
0808 04:37:48.535180 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:48.535262 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:48.536843 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:48.537540 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:48.537548 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:48.537955 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:48.538458 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:48.538473 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.539450 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.539453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:48.540345 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:48.540355 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.540965 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.540970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:48.541234 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:48.542012 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:48.542041 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:48.544529 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:48.546890 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:48.547211 (+ 321us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:48.547234 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:48.547582 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:48.547953 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:48.547957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:48.547959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:48.548230 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:48.548771 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:48.548779 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:48.549048 (+ 269us) client_negotiation.cc:769] Sending connection context
0808 04:37:48.549253 (+ 205us) client_negotiation.cc:240] Negotiation successful
0808 04:37:48.549640 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":137,"threads_started":1}
I20260808 04:37:48.552605 3846 heartbeater.cc:347] Connected to a master server at 127.15.228.126:34873
I20260808 04:37:48.552893 3846 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:48.553422 3846 heartbeater.cc:511] Master 127.15.228.126:34873 requested a full tablet report, sending...
I20260808 04:37:48.554502 3212 ts_manager.cc:194] Registered new tserver with Master: bedaa4b0188e4673a4cefc4dca0d48ea (127.15.228.67:34349)
I20260808 04:37:48.555171 3212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:60453
I20260808 04:37:48.558562 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:37:48.570603 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:48.560882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.15.228.126:34873)
0808 04:37:48.561073 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:48.561079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:48.561230 (+ 151us) server_negotiation.cc:407] Connection header received
0808 04:37:48.561383 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:48.561386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:48.561464 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:48.561552 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:48.562290 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.562811 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.564045 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.564362 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.564546 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:48.566419 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:48.566423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:48.566425 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:48.566454 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:48.568565 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:48.569057 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:48.569061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:48.569063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:48.569158 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:48.569456 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:48.569461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:48.569463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:48.569850 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:48.570021 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:48.570289 (+ 268us) server_negotiation.cc:299] Negotiation successful
0808 04:37:48.570434 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260808 04:37:48.574064 3212 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:37:48.577432 3212 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:37:48.594563 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:48.588713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39810 (local address 127.15.228.66:33473)
0808 04:37:48.589876 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:48.589889 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:48.589908 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:37:48.589984 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:48.589992 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:48.590174 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:48.590349 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:37:48.590556 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.591538 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.593238 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.593806 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.593947 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:48.594104 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:48.594294 (+ 190us) server_negotiation.cc:299] Negotiation successful
0808 04:37:48.594392 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1065,"thread_start_us":125,"threads_started":1}
I20260808 04:37:48.594556 3866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:48.588158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:33473 (local address 127.0.0.1:39810)
0808 04:37:48.588727 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:48.588779 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:48.588924 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:48.590322 (+ 1398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:48.590325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:48.590342 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:37:48.590438 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:48.590444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.591705 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.591708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:48.593112 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:48.593122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.593951 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.593955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:48.594081 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:48.594123 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 04:37:48.594262 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 04:37:48.594390 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":150,"thread_start_us":102,"threads_started":1}
I20260808 04:37:48.596421 3872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:48.588931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:34349 (local address 127.0.0.1:60266)
0808 04:37:48.589737 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:48.589751 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:48.589922 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:48.590550 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:48.590553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:48.590572 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:37:48.590668 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:48.590674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.592002 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.592005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:48.593705 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:48.593736 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.595980 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.595984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:48.596081 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:48.596114 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 04:37:48.596193 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 04:37:48.596273 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":172,"thread_start_us":93,"threads_started":1}
I20260808 04:37:48.596452 3874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:48.589347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60266 (local address 127.15.228.67:34349)
0808 04:37:48.589667 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:48.589680 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:48.589846 (+ 166us) server_negotiation.cc:407] Connection header received
0808 04:37:48.590063 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:48.590067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:48.590231 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:48.590349 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:37:48.590795 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.591881 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.594756 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.595643 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.595751 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:48.595908 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:48.596226 (+ 318us) server_negotiation.cc:299] Negotiation successful
0808 04:37:48.596319 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":117,"threads_started":1}
I20260808 04:37:48.596740 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:48.588603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:40511 (local address 127.0.0.1:42960)
0808 04:37:48.589159 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:48.589173 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:48.589256 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:48.590034 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:48.590042 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:48.590065 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:37:48.590207 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:48.590213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.591822 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.591825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:48.593705 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:48.593736 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.596320 (+ 2584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.596324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:48.596443 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:48.596469 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 04:37:48.596524 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 04:37:48.596596 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":191,"thread_start_us":109,"threads_started":1}
I20260808 04:37:48.597818 3873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:48.589045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42960 (local address 127.15.228.65:40511)
0808 04:37:48.589534 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:48.589541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:48.589566 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:37:48.589645 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:48.589650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:48.589861 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:48.590370 (+ 509us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:37:48.590517 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.591703 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.595107 (+ 3404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.596151 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.597370 (+ 1219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:48.597523 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:48.597575 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 04:37:48.597672 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":155,"threads_started":1}
I20260808 04:37:48.598465 3579 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0310e659cd4aa78800a896ea54aec3 (DEFAULT_TABLE table=test-table [id=4093026be84a4d62af3e73e8649f17b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:37:48.598855 3763 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0310e659cd4aa78800a896ea54aec3 (DEFAULT_TABLE table=test-table [id=4093026be84a4d62af3e73e8649f17b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:37:48.599742 3579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0310e659cd4aa78800a896ea54aec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:37:48.600023 3763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0310e659cd4aa78800a896ea54aec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:37:48.600279 3381 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0310e659cd4aa78800a896ea54aec3 (DEFAULT_TABLE table=test-table [id=4093026be84a4d62af3e73e8649f17b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:37:48.601235 3381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0310e659cd4aa78800a896ea54aec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:37:48.606272 3876 tablet_bootstrap.cc:492] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea: Bootstrap starting.
I20260808 04:37:48.607981 3877 tablet_bootstrap.cc:492] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a: Bootstrap starting.
I20260808 04:37:48.608919 3876 tablet_bootstrap.cc:654] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea: Neither blocks nor log segments found. Creating new log.
I20260808 04:37:48.610507 3876 log.cc:824] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea: Log is configured to *not* fsync() on all Append() calls
I20260808 04:37:48.610606 3877 tablet_bootstrap.cc:654] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a: Neither blocks nor log segments found. Creating new log.
I20260808 04:37:48.611168 3878 tablet_bootstrap.cc:492] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03: Bootstrap starting.
I20260808 04:37:48.611660 3877 log.cc:824] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:37:48.613281 3876 tablet_bootstrap.cc:492] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea: No bootstrap required, opened a new log
I20260808 04:37:48.613510 3876 ts_tablet_manager.cc:1397] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:37:48.613956 3877 tablet_bootstrap.cc:492] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a: No bootstrap required, opened a new log
I20260808 04:37:48.614132 3877 ts_tablet_manager.cc:1397] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:37:48.614336 3878 tablet_bootstrap.cc:654] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03: Neither blocks nor log segments found. Creating new log.
I20260808 04:37:48.615360 3878 log.cc:824] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03: Log is configured to *not* fsync() on all Append() calls
I20260808 04:37:48.617237 3878 tablet_bootstrap.cc:492] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03: No bootstrap required, opened a new log
I20260808 04:37:48.617758 3878 ts_tablet_manager.cc:1397] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260808 04:37:48.617712 3876 raft_consensus.cc:348] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:48.618013 3876 raft_consensus.cc:374] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:37:48.618084 3876 raft_consensus.cc:729] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bedaa4b0188e4673a4cefc4dca0d48ea, State: Initialized, Role: FOLLOWER
I20260808 04:37:48.617919 3877 raft_consensus.cc:348] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:48.618160 3877 raft_consensus.cc:374] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:37:48.618213 3877 raft_consensus.cc:729] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5fcc7345b6c4ccaa21f38c606ada79a, State: Initialized, Role: FOLLOWER
I20260808 04:37:48.618583 3877 consensus_queue.cc:260] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:48.618583 3876 consensus_queue.cc:260] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:48.619303 3846 heartbeater.cc:503] Master 127.15.228.126:34873 was elected leader, sending a full tablet report...
I20260808 04:37:48.619647 3876 ts_tablet_manager.cc:1428] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 04:37:48.619647 3877 ts_tablet_manager.cc:1428] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260808 04:37:48.619962 3667 heartbeater.cc:503] Master 127.15.228.126:34873 was elected leader, sending a full tablet report...
I20260808 04:37:48.624668 3878 raft_consensus.cc:348] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:48.625116 3878 raft_consensus.cc:374] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:37:48.625214 3878 raft_consensus.cc:729] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e1fc50de4bb4098b2b77d3976fbcf03, State: Initialized, Role: FOLLOWER
I20260808 04:37:48.625859 3878 consensus_queue.cc:260] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:48.626770 3473 heartbeater.cc:503] Master 127.15.228.126:34873 was elected leader, sending a full tablet report...
I20260808 04:37:48.627539 3878 ts_tablet_manager.cc:1428] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
W20260808 04:37:48.783124 3848 tablet.cc:2369] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:37:48.784528 3474 tablet.cc:2369] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:37:48.785632 3669 tablet.cc:2369] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:37:48.814666 3886 raft_consensus.cc:482] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:37:48.815008 3886 raft_consensus.cc:504] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:48.816919 3886 leader_election.cc:288] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bedaa4b0188e4673a4cefc4dca0d48ea (127.15.228.67:34349), 3e1fc50de4bb4098b2b77d3976fbcf03 (127.15.228.65:40511)
I20260808 04:37:48.821942 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:48.817288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:34349 (local address 127.15.228.66:48151)
0808 04:37:48.817633 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:48.817653 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:48.817906 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:48.818317 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:48.818323 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:48.818347 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:37:48.818501 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:48.818510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.819516 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.819519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:48.820442 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:48.820452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.821413 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.821417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:48.821527 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:48.821557 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 04:37:48.821646 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 04:37:48.821722 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":28,"thread_start_us":150,"threads_started":1}
I20260808 04:37:48.821954 3874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:48.817474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:48151 (local address 127.15.228.67:34349)
0808 04:37:48.817667 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:48.817673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:48.817796 (+ 123us) server_negotiation.cc:407] Connection header received
0808 04:37:48.818099 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:48.818102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:48.818162 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:48.818260 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:37:48.818742 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.819395 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.820586 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.821185 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.821294 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:48.821341 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:48.821720 (+ 379us) server_negotiation.cc:299] Negotiation successful
0808 04:37:48.821798 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 04:37:48.822536 3873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:48.817952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:55735 (local address 127.15.228.65:40511)
0808 04:37:48.818142 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:48.818147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:48.818159 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:37:48.818210 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:48.818212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:48.818299 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:48.818385 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:37:48.818999 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.819552 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.821095 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.821818 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.821951 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:48.822016 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:48.822251 (+ 235us) server_negotiation.cc:299] Negotiation successful
0808 04:37:48.822416 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 04:37:48.822909 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:48.817450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:40511 (local address 127.15.228.66:55735)
0808 04:37:48.817818 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:48.817834 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:48.817910 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:48.818463 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:48.818466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:48.818491 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:37:48.818764 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:48.818777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.819696 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.819698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:48.820920 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:48.820931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.821961 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.821968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:48.822076 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:48.822106 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 04:37:48.822714 (+ 608us) client_negotiation.cc:240] Negotiation successful
0808 04:37:48.822801 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":219,"thread_start_us":124,"threads_started":1}
I20260808 04:37:48.822919 3788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0310e659cd4aa78800a896ea54aec3" candidate_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" is_pre_election: true
I20260808 04:37:48.823357 3788 raft_consensus.cc:2454] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5fcc7345b6c4ccaa21f38c606ada79a in term 0.
I20260808 04:37:48.823644 3414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0310e659cd4aa78800a896ea54aec3" candidate_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" is_pre_election: true
I20260808 04:37:48.823957 3414 raft_consensus.cc:2454] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5fcc7345b6c4ccaa21f38c606ada79a in term 0.
I20260808 04:37:48.823979 3509 leader_election.cc:302] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bedaa4b0188e4673a4cefc4dca0d48ea, c5fcc7345b6c4ccaa21f38c606ada79a; no voters:
I20260808 04:37:48.824316 3886 raft_consensus.cc:2781] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:37:48.824409 3886 raft_consensus.cc:482] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:37:48.824465 3886 raft_consensus.cc:3037] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:37:48.825594 3886 raft_consensus.cc:504] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:48.826192 3886 leader_election.cc:288] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [CANDIDATE]: Term 1 election: Requested vote from peers bedaa4b0188e4673a4cefc4dca0d48ea (127.15.228.67:34349), 3e1fc50de4bb4098b2b77d3976fbcf03 (127.15.228.65:40511)
I20260808 04:37:48.826385 3788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0310e659cd4aa78800a896ea54aec3" candidate_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea"
I20260808 04:37:48.826457 3414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0310e659cd4aa78800a896ea54aec3" candidate_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03"
I20260808 04:37:48.826592 3414 raft_consensus.cc:3037] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:37:48.826618 3788 raft_consensus.cc:3037] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:37:48.827800 3414 raft_consensus.cc:2454] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5fcc7345b6c4ccaa21f38c606ada79a in term 1.
I20260808 04:37:48.827804 3788 raft_consensus.cc:2454] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5fcc7345b6c4ccaa21f38c606ada79a in term 1.
I20260808 04:37:48.828102 3510 leader_election.cc:302] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e1fc50de4bb4098b2b77d3976fbcf03, c5fcc7345b6c4ccaa21f38c606ada79a; no voters:
I20260808 04:37:48.828298 3886 raft_consensus.cc:2781] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:37:48.828573 3886 raft_consensus.cc:686] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [term 1 LEADER]: Becoming Leader. State: Replica: c5fcc7345b6c4ccaa21f38c606ada79a, State: Running, Role: LEADER
I20260808 04:37:48.828977 3886 consensus_queue.cc:237] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:48.833302 3211 catalog_manager.cc:5495] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a reported cstate change: term changed from 0 to 1, leader changed from <none> to c5fcc7345b6c4ccaa21f38c606ada79a (127.15.228.66). New cstate: current_term: 1 leader_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } health_report { overall_health: UNKNOWN } } }
I20260808 04:37:48.895655 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:48.890583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.15.228.66:33473)
0808 04:37:48.890726 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:48.890730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:48.890827 (+ 97us) server_negotiation.cc:407] Connection header received
0808 04:37:48.891092 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:48.891095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:48.891168 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:48.891288 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:37:48.891765 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.892283 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.893426 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:48.893648 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:48.893750 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:48.894172 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:37:48.895189 (+ 1017us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:37:48.895331 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:48.895471 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 04:37:48.895533 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:37:48.903541 3414 raft_consensus.cc:1261] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Refusing update from remote peer c5fcc7345b6c4ccaa21f38c606ada79a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:37:48.903541 3788 raft_consensus.cc:1261] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea [term 1 FOLLOWER]: Refusing update from remote peer c5fcc7345b6c4ccaa21f38c606ada79a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:37:48.904281 3886 consensus_queue.cc:1035] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:37:48.904546 3894 consensus_queue.cc:1035] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:37:48.913549 3900 mvcc.cc:205] Tried to move back new op lower bound from 7316127207020601344 to 7316127206726205440. Current Snapshot: MvccSnapshot[applied={T|T < 7316127207020601344}]
I20260808 04:37:48.914465 3902 mvcc.cc:205] Tried to move back new op lower bound from 7316127207020601344 to 7316127206726205440. Current Snapshot: MvccSnapshot[applied={T|T < 7316127207020601344}]
I20260808 04:37:48.915469 3905 mvcc.cc:205] Tried to move back new op lower bound from 7316127207020601344 to 7316127206726205440. Current Snapshot: MvccSnapshot[applied={T|T < 7316127207020601344}]
I20260808 04:37:48.926837 3208 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:37:48.927187 3208 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:37:48.932252 3208 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa: Sending DeleteTablet for 3 replicas of tablet 6b0310e659cd4aa78800a896ea54aec3
I20260808 04:37:48.933705 3381 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0310e659cd4aa78800a896ea54aec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:37:48 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:42960
I20260808 04:37:48.934577 3917 tablet_replica.cc:330] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03: stopping tablet replica
I20260808 04:37:48.934994 3917 raft_consensus.cc:2229] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:37:48.935388 3917 raft_consensus.cc:2258] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:37:48.939793 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 3297
I20260808 04:37:48.939743 3763 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0310e659cd4aa78800a896ea54aec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:37:48 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:60266
I20260808 04:37:48.942725 3579 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0310e659cd4aa78800a896ea54aec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:37:48 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:39810
I20260808 04:37:48.943637 3919 tablet_replica.cc:330] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a: stopping tablet replica
I20260808 04:37:48.944121 3919 raft_consensus.cc:2229] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:37:48.944625 3919 raft_consensus.cc:2258] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:37:48.948082 3919 ts_tablet_manager.cc:1905] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:37:48.951038 3919 ts_tablet_manager.cc:1918] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:37:48.951145 3919 log.cc:1196] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/6b0310e659cd4aa78800a896ea54aec3
I20260808 04:37:48.951470 3919 ts_tablet_manager.cc:1939] T 6b0310e659cd4aa78800a896ea54aec3 P c5fcc7345b6c4ccaa21f38c606ada79a: Deleting consensus metadata
I20260808 04:37:48.951983 3918 tablet_replica.cc:330] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea: stopping tablet replica
I20260808 04:37:48.953001 3192 catalog_manager.cc:4841] TS c5fcc7345b6c4ccaa21f38c606ada79a (127.15.228.66:33473): tablet 6b0310e659cd4aa78800a896ea54aec3 (table test-table [id=4093026be84a4d62af3e73e8649f17b1]) successfully deleted
I20260808 04:37:48.952533 3918 raft_consensus.cc:2229] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea [term 1 FOLLOWER]: Raft consensus shutting down.
W20260808 04:37:48.954911 3193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.228.65:40511 (error 108)
I20260808 04:37:48.955360 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 3481
I20260808 04:37:48.958678 3918 raft_consensus.cc:2258] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:37:48.962606 3872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:48.962166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:40511 (local address 127.0.0.1:42972)
0808 04:37:48.962397 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:48.962483 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.65:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260808 04:37:48.962821 3193 catalog_manager.cc:4584] TS 3e1fc50de4bb4098b2b77d3976fbcf03 (127.15.228.65:40511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b0310e659cd4aa78800a896ea54aec3: Network error: Client connection negotiation failed: client connection to 127.15.228.65:40511: connect: Connection refused (error 111)
I20260808 04:37:48.963609 3918 ts_tablet_manager.cc:1905] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:37:48.966157 3918 ts_tablet_manager.cc:1918] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:37:48.966284 3918 log.cc:1196] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/6b0310e659cd4aa78800a896ea54aec3
I20260808 04:37:48.966678 3918 ts_tablet_manager.cc:1939] T 6b0310e659cd4aa78800a896ea54aec3 P bedaa4b0188e4673a4cefc4dca0d48ea: Deleting consensus metadata
I20260808 04:37:48.968305 3192 catalog_manager.cc:4841] TS bedaa4b0188e4673a4cefc4dca0d48ea (127.15.228.67:34349): tablet 6b0310e659cd4aa78800a896ea54aec3 (table test-table [id=4093026be84a4d62af3e73e8649f17b1]) successfully deleted
I20260808 04:37:48.969496 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 3673
I20260808 04:37:48.984560 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 3169
I20260808 04:37:48.999338 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:34873
--webserver_interface=127.15.228.126
--webserver_port=32915
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:34873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:37:49.130877 3922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:49.131174 3922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:49.131218 3922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:49.135144 3922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:37:49.135197 3922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:49.135217 3922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:49.135234 3922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:37:49.135262 3922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:37:49.141058 3922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:34873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:34873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=32915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:49.142191 3922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:49.143394 3922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:49.161207 3933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:49.161211 3937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:49.161320 3922 server_base.cc:1034] running on GCE node
W20260808 04:37:49.161207 3932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:49.162196 3922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:49.163784 3922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:49.165132 3922 hybrid_clock.cc:648] HybridClock initialized: now 1786163869165091 us; error 56 us; skew 500 ppm
Aug 08 04:37:49 dist-test-slave-drl0 krb5kdc[3152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163869, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:49.168342 3922 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:37:49.169386 3922 webserver.cc:460] Webserver started at http://127.15.228.126:32915/ using document root <none> and password file <none>
I20260808 04:37:49.170019 3922 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:49.170101 3922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:37:49.175930 3922 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:37:49.179284 3947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:49.180977 3922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:37:49.181188 3922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "53d25bfe01ec48abb6226f2b1aef9afa"
format_stamp: "Formatted at 2026-08-08 04:37:47 on dist-test-slave-drl0"
server_key: "bf07bf9316c10744bf43fb6ee7a8cd26"
server_key_iv: "40fa369efbf0a463793da8f15502ba74"
server_key_version: "encryptionkey@0"
I20260808 04:37:49.181882 3922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:49.193265 3922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:49.197717 3922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:49.197976 3922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:49.206369 3922 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:34873
I20260808 04:37:49.206696 4019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:34873 every 8 connection(s)
I20260808 04:37:49.207605 3922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:37:49.208992 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 3922
I20260808 04:37:49.209626 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:40511
--local_ip_for_outbound_sockets=127.15.228.65
--tserver_master_addrs=127.15.228.126:34873
--webserver_port=42241
--webserver_interface=127.15.228.65
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:37:49.214745 4021 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:37:49.216957 4021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa: Bootstrap starting.
I20260808 04:37:49.239229 4021 log.cc:824] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa: Log is configured to *not* fsync() on all Append() calls
I20260808 04:37:49.251031 4021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:37:49.251572 4021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa: Bootstrap complete.
I20260808 04:37:49.256973 4021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34873 } }
I20260808 04:37:49.257419 4021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d25bfe01ec48abb6226f2b1aef9afa, State: Initialized, Role: FOLLOWER
I20260808 04:37:49.257939 4021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34873 } }
I20260808 04:37:49.258059 4021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:37:49.258158 4021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:37:49.258276 4021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:37:49.260552 4021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34873 } }
I20260808 04:37:49.260907 4021 leader_election.cc:302] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53d25bfe01ec48abb6226f2b1aef9afa; no voters:
I20260808 04:37:49.261250 4021 leader_election.cc:288] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 04:37:49.261461 4025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:37:49.261807 4025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 2 LEADER]: Becoming Leader. State: Replica: 53d25bfe01ec48abb6226f2b1aef9afa, State: Running, Role: LEADER
I20260808 04:37:49.262207 4025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34873 } }
I20260808 04:37:49.262666 4021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:37:49.264003 4027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53d25bfe01ec48abb6226f2b1aef9afa. Latest consensus state: current_term: 2 leader_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34873 } } }
I20260808 04:37:49.264012 4026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34873 } } }
I20260808 04:37:49.264269 4026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [sys.catalog]: This master's current role is: LEADER
I20260808 04:37:49.264307 4027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [sys.catalog]: This master's current role is: LEADER
I20260808 04:37:49.269366 4045 catalog_manager.cc:1259] Loaded cluster ID: 273838deed8842cdb8d537d1ee70de99
I20260808 04:37:49.269441 4045 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa: loading cluster ID for follower catalog manager: success
W20260808 04:37:49.270571 4045 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260808 04:37:49.270659 4045 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260808 04:37:49.270732 4035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:37:49.273260 4035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:37:49.273726 4035 catalog_manager.cc:1259] Loaded cluster ID: 273838deed8842cdb8d537d1ee70de99
I20260808 04:37:49.273790 4035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:37:49.274796 4035 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 04:37:49.279477 4035 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 04:37:49.286499 4035 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:37:49.287428 4035 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa: Loaded TSK: 0
I20260808 04:37:49.288700 4035 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:37:49.351099 4023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:49.351372 4023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:49.351416 4023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:49.355518 4023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:49.355577 4023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:49.355636 4023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:37:49.361769 4023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:40511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=42241
--tserver_master_addrs=127.15.228.126:34873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:49.362856 4023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:49.364108 4023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:49.376243 4056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:49.376346 4063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:49.376576 4058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:49.376997 4023 server_base.cc:1034] running on GCE node
I20260808 04:37:49.377413 4023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:49.378103 4023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:49.379331 4023 hybrid_clock.cc:648] HybridClock initialized: now 1786163869379301 us; error 42 us; skew 500 ppm
Aug 08 04:37:49 dist-test-slave-drl0 krb5kdc[3152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163869, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:49.382241 4023 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:37:49.383486 4023 webserver.cc:460] Webserver started at http://127.15.228.65:42241/ using document root <none> and password file <none>
I20260808 04:37:49.384090 4023 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:49.384174 4023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:37:49.388622 4023 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:37:49.391176 4072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:49.392529 4023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:37:49.392632 4023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "3e1fc50de4bb4098b2b77d3976fbcf03"
format_stamp: "Formatted at 2026-08-08 04:37:47 on dist-test-slave-drl0"
server_key: "34b67d3a73530ee9e7342b88cc1968e0"
server_key_iv: "43f533ec3d898e5673d0ce952e642c74"
server_key_version: "encryptionkey@0"
I20260808 04:37:49.393188 4023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:49.416841 4023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:49.420682 4023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:49.420876 4023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:49.421571 4023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:37:49.422888 4082 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 04:37:49.428422 4023 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 04:37:49.428526 4023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260808 04:37:49.428618 4023 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 04:37:49.430469 4023 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 04:37:49.430536 4023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 04:37:49.430802 4082 tablet_bootstrap.cc:492] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03: Bootstrap starting.
I20260808 04:37:49.438781 4082 log.cc:824] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03: Log is configured to *not* fsync() on all Append() calls
I20260808 04:37:49.442291 4023 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:40511
I20260808 04:37:49.442358 4241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:40511 every 8 connection(s)
I20260808 04:37:49.443241 4023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:37:49.449687 4082 tablet_bootstrap.cc:492] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:37:49.450207 4082 tablet_bootstrap.cc:492] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03: Bootstrap complete.
I20260808 04:37:49.450481 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 4023
I20260808 04:37:49.450650 4082 ts_tablet_manager.cc:1397] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.003s
I20260808 04:37:49.450985 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:33473
--local_ip_for_outbound_sockets=127.15.228.66
--tserver_master_addrs=127.15.228.126:34873
--webserver_port=42265
--webserver_interface=127.15.228.66
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:37:49.455138 4082 raft_consensus.cc:348] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:49.455932 4082 raft_consensus.cc:729] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e1fc50de4bb4098b2b77d3976fbcf03, State: Initialized, Role: FOLLOWER
I20260808 04:37:49.456681 4082 consensus_queue.cc:260] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:49.458276 4082 ts_tablet_manager.cc:1428] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
Aug 08 04:37:49 dist-test-slave-drl0 krb5kdc[3152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163869, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:37:49.467008 4251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:49.445593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:42189 (local address 127.15.228.126:34873)
0808 04:37:49.446082 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:49.446102 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:49.446449 (+ 347us) server_negotiation.cc:407] Connection header received
0808 04:37:49.448219 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:49.448230 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:49.448557 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:49.448812 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:49.450507 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.452059 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.454829 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.455764 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.456036 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:49.460693 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:49.460701 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:49.460718 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:49.460760 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:49.462875 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:49.463919 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:49.463926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:49.463933 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:49.464103 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:49.464550 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:49.464555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:49.464561 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:49.465013 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:49.465167 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:49.465733 (+ 566us) server_negotiation.cc:299] Negotiation successful
0808 04:37:49.466019 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":122,"threads_started":1}
I20260808 04:37:49.467020 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:49.445367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:42189)
0808 04:37:49.446113 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:49.446199 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:49.447759 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:49.448946 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:49.448955 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:49.449546 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:49.450270 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:49.450295 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.452552 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.452562 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:49.454581 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:49.454600 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.456117 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.456124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:49.456384 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:49.457976 (+ 1592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:49.458005 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:49.460340 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:49.463097 (+ 2757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:49.463104 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:49.463122 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:49.463736 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:49.464232 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:49.464235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:49.464237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:49.464425 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:49.465145 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:49.465152 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:49.465419 (+ 267us) client_negotiation.cc:769] Sending connection context
0808 04:37:49.465620 (+ 201us) client_negotiation.cc:240] Negotiation successful
0808 04:37:49.466010 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":148,"threads_started":1}
I20260808 04:37:49.469868 4246 heartbeater.cc:347] Connected to a master server at 127.15.228.126:34873
I20260808 04:37:49.470140 4246 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:49.470664 4246 heartbeater.cc:511] Master 127.15.228.126:34873 requested a full tablet report, sending...
I20260808 04:37:49.473951 3969 ts_manager.cc:194] Registered new tserver with Master: 3e1fc50de4bb4098b2b77d3976fbcf03 (127.15.228.65:40511)
I20260808 04:37:49.478246 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:42189
I20260808 04:37:49.481657 4246 heartbeater.cc:503] Master 127.15.228.126:34873 was elected leader, sending a full tablet report...
Aug 08 04:37:49 dist-test-slave-drl0 krb5kdc[3152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163869, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for kudu/127.15.228.65@KRBTEST.COM
I20260808 04:37:49.490671 4256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:49.478110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42980 (local address 127.15.228.65:40511)
0808 04:37:49.478819 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:49.478832 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:49.479379 (+ 547us) server_negotiation.cc:407] Connection header received
0808 04:37:49.479694 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:49.479700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:49.479916 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:49.480081 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:49.481008 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.482183 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.483500 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.483982 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.484117 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:49.486836 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:49.486846 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:49.486860 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:49.486886 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:49.488679 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:49.489179 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:49.489185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:49.489187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:49.489273 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:49.489568 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:49.489572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:49.489574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:49.489811 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:49.489968 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:49.490311 (+ 343us) server_negotiation.cc:299] Negotiation successful
0808 04:37:49.490457 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":127,"threads_started":1}
I20260808 04:37:49.490847 4255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:49.477965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:40511 (local address 127.0.0.1:42980)
0808 04:37:49.479222 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:49.479285 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:49.479461 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:49.480137 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:49.480143 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:49.480678 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:49.480847 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:49.480857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.482327 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.482331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:49.483341 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:49.483357 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.484124 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.484128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:49.484263 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:49.484886 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:49.484899 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:49.486657 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:49.488804 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:49.488814 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:49.488819 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:49.489056 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:49.489373 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:49.489377 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:49.489378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:49.489472 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:49.489933 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:49.489939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:49.490074 (+ 135us) client_negotiation.cc:769] Sending connection context
0808 04:37:49.490217 (+ 143us) client_negotiation.cc:240] Negotiation successful
0808 04:37:49.490456 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"thread_start_us":180,"threads_started":1}
I20260808 04:37:49.491907 4143 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0310e659cd4aa78800a896ea54aec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:37:48 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:42980
I20260808 04:37:49.492928 4258 tablet_replica.cc:330] stopping tablet replica
I20260808 04:37:49.493274 4258 raft_consensus.cc:2229] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:37:49.493582 4258 raft_consensus.cc:2258] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:37:49.494772 4258 ts_tablet_manager.cc:1905] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:37:49.500345 4258 ts_tablet_manager.cc:1918] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:37:49.500483 4258 log.cc:1196] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/wal/wals/6b0310e659cd4aa78800a896ea54aec3
I20260808 04:37:49.500784 4258 ts_tablet_manager.cc:1939] T 6b0310e659cd4aa78800a896ea54aec3 P 3e1fc50de4bb4098b2b77d3976fbcf03: Deleting consensus metadata
I20260808 04:37:49.503019 3949 catalog_manager.cc:4841] TS 3e1fc50de4bb4098b2b77d3976fbcf03 (127.15.228.65:40511): tablet 6b0310e659cd4aa78800a896ea54aec3 (table test-table [id=4093026be84a4d62af3e73e8649f17b1]) successfully deleted
W20260808 04:37:49.593482 4252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:49.593755 4252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:49.593945 4252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:49.598001 4252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:49.598063 4252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:49.598125 4252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:37:49.603552 4252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:33473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=42265
--tserver_master_addrs=127.15.228.126:34873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:49.604696 4252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:49.606560 4252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:49.617457 4272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:49.617503 4252 server_base.cc:1034] running on GCE node
W20260808 04:37:49.617471 4274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:49.617516 4271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:49.618597 4252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:49.619401 4252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:49.620708 4252 hybrid_clock.cc:648] HybridClock initialized: now 1786163869620632 us; error 76 us; skew 500 ppm
Aug 08 04:37:49 dist-test-slave-drl0 krb5kdc[3152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163869, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:49.624651 4252 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:37:49.626057 4252 webserver.cc:460] Webserver started at http://127.15.228.66:42265/ using document root <none> and password file <none>
I20260808 04:37:49.627065 4252 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:49.627218 4252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:37:49.632666 4252 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:37:49.635383 4281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:49.636957 4252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:37:49.637120 4252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "c5fcc7345b6c4ccaa21f38c606ada79a"
format_stamp: "Formatted at 2026-08-08 04:37:48 on dist-test-slave-drl0"
server_key: "c8f9044eb78ee1829f7cd8d933e611d8"
server_key_iv: "eb154396f3a905ba580482b27ec47501"
server_key_version: "encryptionkey@0"
I20260808 04:37:49.637763 4252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:49.656585 4252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:49.662046 4252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:49.662294 4252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:49.663106 4252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:37:49.664177 4252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:37:49.664245 4252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:49.664315 4252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:37:49.664345 4252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:49.678421 4252 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:33473
I20260808 04:37:49.679872 4252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:37:49.680516 4437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:33473 every 8 connection(s)
I20260808 04:37:49.682252 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 4252
I20260808 04:37:49.682817 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:34349
--local_ip_for_outbound_sockets=127.15.228.67
--tserver_master_addrs=127.15.228.126:34873
--webserver_port=39225
--webserver_interface=127.15.228.67
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:37:49 dist-test-slave-drl0 krb5kdc[3152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163869, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:37:49.699263 4251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:49.682223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:46685 (local address 127.15.228.126:34873)
0808 04:37:49.682455 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:49.682461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:49.683312 (+ 851us) server_negotiation.cc:407] Connection header received
0808 04:37:49.685093 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:49.685100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:49.685168 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:49.685279 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:49.687175 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.687768 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.689859 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.690128 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.690228 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:49.694182 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:49.694188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:49.694191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:49.694219 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:49.696291 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:49.696980 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:49.696987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:49.696991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:49.697094 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:49.697802 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:49.697809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:49.697814 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:49.698106 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:49.698237 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:49.698906 (+ 669us) server_negotiation.cc:299] Negotiation successful
0808 04:37:49.699078 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 04:37:49.700265 4444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:49.682575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.66:46685)
0808 04:37:49.683143 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:49.683203 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:49.684771 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:49.685485 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:49.685498 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:49.686101 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:49.686881 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:49.686906 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.687918 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.687925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:49.689697 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:49.689714 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.690327 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.690334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:49.690613 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:49.691534 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:49.691564 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:49.693799 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:49.696537 (+ 2738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:49.696543 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:49.696564 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:49.696847 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:49.697232 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:49.697235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:49.697236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:49.697660 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:49.698268 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:49.698274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:49.698625 (+ 351us) client_negotiation.cc:769] Sending connection context
0808 04:37:49.698810 (+ 185us) client_negotiation.cc:240] Negotiation successful
0808 04:37:49.699164 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":164,"threads_started":1}
I20260808 04:37:49.701884 4438 heartbeater.cc:347] Connected to a master server at 127.15.228.126:34873
I20260808 04:37:49.702172 4438 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:49.702718 4438 heartbeater.cc:511] Master 127.15.228.126:34873 requested a full tablet report, sending...
I20260808 04:37:49.703929 3969 ts_manager.cc:194] Registered new tserver with Master: c5fcc7345b6c4ccaa21f38c606ada79a (127.15.228.66:33473)
I20260808 04:37:49.704818 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:46685
W20260808 04:37:49.822427 4445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:49.822690 4445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:49.822733 4445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:49.827986 4445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:49.828104 4445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:49.828197 4445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:37:49.834836 4445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:34349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=39225
--tserver_master_addrs=127.15.228.126:34873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:49.836315 4445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:49.837446 4445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:49.848791 4459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:49.848879 4445 server_base.cc:1034] running on GCE node
W20260808 04:37:49.848805 4457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:49.848799 4462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:49.849665 4445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:49.850306 4445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:49.853617 4445 hybrid_clock.cc:648] HybridClock initialized: now 1786163869853467 us; error 180 us; skew 500 ppm
Aug 08 04:37:49 dist-test-slave-drl0 krb5kdc[3152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163869, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:49.856638 4445 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:37:49.857528 4445 webserver.cc:460] Webserver started at http://127.15.228.67:39225/ using document root <none> and password file <none>
I20260808 04:37:49.858075 4445 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:49.858150 4445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:37:49.862351 4445 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:37:49.865106 4472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:49.866863 4445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:37:49.867069 4445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "bedaa4b0188e4673a4cefc4dca0d48ea"
format_stamp: "Formatted at 2026-08-08 04:37:48 on dist-test-slave-drl0"
server_key: "73b4dcaa4656c17d741931a8ee75af6e"
server_key_iv: "aebd2605e82b9df8ba210b22dabb2001"
server_key_version: "encryptionkey@0"
I20260808 04:37:49.867709 4445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:49.880012 4445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:49.883764 4445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:49.883988 4445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:49.884636 4445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:37:49.885514 4445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:37:49.885599 4445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:49.885692 4445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:37:49.885735 4445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:49.896446 4445 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:34349
I20260808 04:37:49.896514 4630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:34349 every 8 connection(s)
I20260808 04:37:49.897670 4445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:37:49.903975 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 4445
Aug 08 04:37:49 dist-test-slave-drl0 krb5kdc[3152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163869, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:37:49.914515 4251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:49.899760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:44525 (local address 127.15.228.126:34873)
0808 04:37:49.899928 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:49.899932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:49.900818 (+ 886us) server_negotiation.cc:407] Connection header received
0808 04:37:49.902292 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:49.902296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:49.902386 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:49.902518 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:49.903885 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.904490 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.905992 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.906207 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.906294 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:49.909818 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:49.909823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:49.909827 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:49.909859 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:49.911532 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:49.912214 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:49.912220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:49.912225 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:49.912408 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:49.912905 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:49.912909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:49.912910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:49.913189 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:49.913358 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:49.914188 (+ 830us) server_negotiation.cc:299] Negotiation successful
0808 04:37:49.914328 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:37:49.914757 4636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:49.905008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38206 (local address 127.15.228.126:34873)
0808 04:37:49.905363 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:49.905368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:49.905572 (+ 204us) server_negotiation.cc:407] Connection header received
0808 04:37:49.905711 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:49.905714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:49.905768 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:49.905844 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:49.906821 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.907612 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.908730 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.909166 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.909270 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:49.910923 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:49.910973 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:49.910976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:49.911024 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:49.913027 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:49.913468 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:49.913471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:49.913473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:49.913559 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:49.913847 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:49.913850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:49.913852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:49.914045 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:49.914270 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:49.914454 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 04:37:49.914573 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":107,"threads_started":1}
I20260808 04:37:49.915563 4635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:49.900128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.67:44525)
0808 04:37:49.900653 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:49.900696 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:49.901960 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:49.902702 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:49.902709 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:49.903193 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:49.903701 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:49.903715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.904624 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.904628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:49.905840 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:49.905854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.906384 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.906391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:49.906601 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:49.907421 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:49.907438 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:49.909596 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:49.911668 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:49.911678 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:49.911696 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:49.912062 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:49.912540 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:49.912543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:49.912545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:49.912749 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:49.913389 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:49.913400 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:49.913885 (+ 485us) client_negotiation.cc:769] Sending connection context
0808 04:37:49.914122 (+ 237us) client_negotiation.cc:240] Negotiation successful
0808 04:37:49.914474 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":123,"threads_started":1}
I20260808 04:37:49.916998 4631 heartbeater.cc:347] Connected to a master server at 127.15.228.126:34873
I20260808 04:37:49.917248 4631 heartbeater.cc:464] Registering TS with master...
I20260808 04:37:49.917757 4631 heartbeater.cc:511] Master 127.15.228.126:34873 requested a full tablet report, sending...
I20260808 04:37:49.918810 3968 ts_manager.cc:194] Registered new tserver with Master: bedaa4b0188e4673a4cefc4dca0d48ea (127.15.228.67:34349)
I20260808 04:37:49.919466 3968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:44525
I20260808 04:37:49.929966 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:37:49.935777 4636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:49.931263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38214 (local address 127.15.228.126:34873)
0808 04:37:49.931471 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:49.931475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:49.931627 (+ 152us) server_negotiation.cc:407] Connection header received
0808 04:37:49.931782 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:49.931784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:49.931891 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:49.931979 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:37:49.932373 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.933024 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.934190 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.934453 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.934550 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:49.934964 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:37:49.935392 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:37:49.935500 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:49.935603 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 04:37:49.935663 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260808 04:37:49.936551 3968 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:37:49.940287 3968 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:37:49.950405 4143 tablet_service.cc:1467] Processing CreateTablet for tablet 668a4f57693e4d9d905a4aa6c5cdd38f (DEFAULT_TABLE table=test-table [id=b7921be964d74b93ae04882392746949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:37:49.951167 4143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668a4f57693e4d9d905a4aa6c5cdd38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:37:49.954103 4082 tablet_bootstrap.cc:492] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03: Bootstrap starting.
I20260808 04:37:49.955240 4082 tablet_bootstrap.cc:654] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03: Neither blocks nor log segments found. Creating new log.
I20260808 04:37:49.955742 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:49.949594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:34349 (local address 127.0.0.1:60274)
0808 04:37:49.950008 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:49.950022 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:49.950113 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:49.951143 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:49.951146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:49.951165 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:37:49.951287 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:49.951298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.952645 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.952650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:49.954127 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:49.954140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.955269 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.955273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:49.955369 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:49.955414 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 04:37:49.955508 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 04:37:49.955589 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":55,"thread_start_us":101,"threads_started":1}
I20260808 04:37:49.955754 4644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:49.950065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60274 (local address 127.15.228.67:34349)
0808 04:37:49.950681 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:49.950696 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:49.950717 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:37:49.950794 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:49.950799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:49.950992 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:49.951166 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:37:49.951422 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.952513 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.954295 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.955143 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.955278 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:49.955436 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:49.955541 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 04:37:49.955631 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":196,"threads_started":1}
I20260808 04:37:49.956215 4255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:49.949286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:33473 (local address 127.0.0.1:39836)
0808 04:37:49.949452 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:49.949469 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:49.949554 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:49.950748 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:49.950751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:49.950770 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:37:49.950892 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:49.950899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.952807 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.952810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:49.954127 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:49.954140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.955574 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.955578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:49.955909 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:49.955937 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:37:49.956004 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 04:37:49.956099 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":45}
I20260808 04:37:49.956868 4643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:49.949523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39836 (local address 127.15.228.66:33473)
0808 04:37:49.950206 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:49.950220 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:49.950245 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:37:49.950328 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:49.950334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:49.950563 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:49.951089 (+ 526us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:37:49.951264 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.952666 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.954523 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:49.955385 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:49.956399 (+ 1014us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:49.956568 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:49.956623 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 04:37:49.956719 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":168,"threads_started":1}
I20260808 04:37:49.957468 4082 tablet_bootstrap.cc:492] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03: No bootstrap required, opened a new log
I20260808 04:37:49.957559 4082 ts_tablet_manager.cc:1397] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:37:49.957976 4082 raft_consensus.cc:348] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:49.958098 4082 raft_consensus.cc:374] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:37:49.958145 4082 raft_consensus.cc:729] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e1fc50de4bb4098b2b77d3976fbcf03, State: Initialized, Role: FOLLOWER
I20260808 04:37:49.958302 4082 consensus_queue.cc:260] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:49.958717 4082 ts_tablet_manager.cc:1428] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:37:49.959640 4539 tablet_service.cc:1467] Processing CreateTablet for tablet 668a4f57693e4d9d905a4aa6c5cdd38f (DEFAULT_TABLE table=test-table [id=b7921be964d74b93ae04882392746949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:37:49.959640 4351 tablet_service.cc:1467] Processing CreateTablet for tablet 668a4f57693e4d9d905a4aa6c5cdd38f (DEFAULT_TABLE table=test-table [id=b7921be964d74b93ae04882392746949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:37:49.960790 4351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668a4f57693e4d9d905a4aa6c5cdd38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:37:49.960809 4539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668a4f57693e4d9d905a4aa6c5cdd38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:37:49.969694 4649 tablet_bootstrap.cc:492] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea: Bootstrap starting.
I20260808 04:37:49.971486 4650 tablet_bootstrap.cc:492] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a: Bootstrap starting.
I20260808 04:37:49.974406 4650 tablet_bootstrap.cc:654] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a: Neither blocks nor log segments found. Creating new log.
I20260808 04:37:49.975585 4650 log.cc:824] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:37:49.977445 4649 tablet_bootstrap.cc:654] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea: Neither blocks nor log segments found. Creating new log.
I20260808 04:37:49.978118 4649 log.cc:824] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea: Log is configured to *not* fsync() on all Append() calls
I20260808 04:37:49.978956 4650 tablet_bootstrap.cc:492] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a: No bootstrap required, opened a new log
I20260808 04:37:49.979213 4650 ts_tablet_manager.cc:1397] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 04:37:49.982035 4649 tablet_bootstrap.cc:492] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea: No bootstrap required, opened a new log
I20260808 04:37:49.982262 4649 ts_tablet_manager.cc:1397] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20260808 04:37:49.984108 4650 raft_consensus.cc:348] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:49.984411 4650 raft_consensus.cc:374] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:37:49.984484 4650 raft_consensus.cc:729] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5fcc7345b6c4ccaa21f38c606ada79a, State: Initialized, Role: FOLLOWER
I20260808 04:37:49.984906 4650 consensus_queue.cc:260] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:49.985675 4438 heartbeater.cc:503] Master 127.15.228.126:34873 was elected leader, sending a full tablet report...
I20260808 04:37:49.986028 4650 ts_tablet_manager.cc:1428] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 04:37:49.986773 4649 raft_consensus.cc:348] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:49.987227 4649 raft_consensus.cc:374] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:37:49.987300 4649 raft_consensus.cc:729] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bedaa4b0188e4673a4cefc4dca0d48ea, State: Initialized, Role: FOLLOWER
I20260808 04:37:49.988394 4649 consensus_queue.cc:260] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:49.989370 4631 heartbeater.cc:503] Master 127.15.228.126:34873 was elected leader, sending a full tablet report...
I20260808 04:37:49.989737 4649 ts_tablet_manager.cc:1428] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260808 04:37:50.080271 4648 raft_consensus.cc:482] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:37:50.080628 4648 raft_consensus.cc:504] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:50.081962 4648 leader_election.cc:288] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5fcc7345b6c4ccaa21f38c606ada79a (127.15.228.66:33473), bedaa4b0188e4673a4cefc4dca0d48ea (127.15.228.67:34349)
I20260808 04:37:50.086597 4643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:50.082562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:55113 (local address 127.15.228.66:33473)
0808 04:37:50.082726 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:50.082737 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:50.083033 (+ 296us) server_negotiation.cc:407] Connection header received
0808 04:37:50.083384 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:50.083387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:50.083461 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:50.083520 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:37:50.083894 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:50.084409 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:50.085553 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:50.086009 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:50.086139 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:50.086194 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:50.086396 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 04:37:50.086465 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:37:50.086639 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:50.082414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:33473 (local address 127.15.228.65:55113)
0808 04:37:50.082876 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:50.082895 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:50.083067 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:50.083657 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:50.083660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:50.083678 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:37:50.083775 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:50.083779 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:50.084548 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:50.084551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:50.085427 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:50.085437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:50.086136 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:50.086139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:50.086233 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:50.086300 (+ 67us) client_negotiation.cc:769] Sending connection context
0808 04:37:50.086380 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 04:37:50.086460 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":230,"threads_started":1}
I20260808 04:37:50.087070 4656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:50.083060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:34349 (local address 127.15.228.65:52741)
0808 04:37:50.083463 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:50.083478 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:50.083624 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:50.083890 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:50.083893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:50.083912 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:37:50.084021 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:50.084033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:50.084968 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:50.084972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:50.085814 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:50.085824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:50.086601 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:50.086605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:50.086719 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:50.086753 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 04:37:50.086830 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 04:37:50.086889 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":196,"threads_started":1}
I20260808 04:37:50.087076 4644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:50.083234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:52741 (local address 127.15.228.67:34349)
0808 04:37:50.083391 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:50.083396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:50.083560 (+ 164us) server_negotiation.cc:407] Connection header received
0808 04:37:50.083739 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:50.083742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:50.083792 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:50.083885 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:37:50.084138 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:50.084813 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:50.085944 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:50.086471 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:50.086605 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:50.086652 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:50.086861 (+ 209us) server_negotiation.cc:299] Negotiation successful
0808 04:37:50.086977 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:37:50.087711 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668a4f57693e4d9d905a4aa6c5cdd38f" candidate_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" is_pre_election: true
I20260808 04:37:50.087777 4566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668a4f57693e4d9d905a4aa6c5cdd38f" candidate_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" is_pre_election: true
I20260808 04:37:50.088058 4374 raft_consensus.cc:2454] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1fc50de4bb4098b2b77d3976fbcf03 in term 0.
I20260808 04:37:50.088080 4566 raft_consensus.cc:2454] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e1fc50de4bb4098b2b77d3976fbcf03 in term 0.
I20260808 04:37:50.088552 4073 leader_election.cc:302] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e1fc50de4bb4098b2b77d3976fbcf03, bedaa4b0188e4673a4cefc4dca0d48ea; no voters:
I20260808 04:37:50.088847 4648 raft_consensus.cc:2781] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:37:50.088938 4648 raft_consensus.cc:482] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:37:50.088971 4648 raft_consensus.cc:3037] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:37:50.090160 4648 raft_consensus.cc:504] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:50.090533 4648 leader_election.cc:288] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 1 election: Requested vote from peers c5fcc7345b6c4ccaa21f38c606ada79a (127.15.228.66:33473), bedaa4b0188e4673a4cefc4dca0d48ea (127.15.228.67:34349)
I20260808 04:37:50.091027 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668a4f57693e4d9d905a4aa6c5cdd38f" candidate_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a"
I20260808 04:37:50.091181 4374 raft_consensus.cc:3037] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:37:50.091324 4566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668a4f57693e4d9d905a4aa6c5cdd38f" candidate_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea"
I20260808 04:37:50.091455 4566 raft_consensus.cc:3037] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:37:50.092705 4374 raft_consensus.cc:2454] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1fc50de4bb4098b2b77d3976fbcf03 in term 1.
I20260808 04:37:50.093005 4566 raft_consensus.cc:2454] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e1fc50de4bb4098b2b77d3976fbcf03 in term 1.
I20260808 04:37:50.093044 4073 leader_election.cc:302] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e1fc50de4bb4098b2b77d3976fbcf03, c5fcc7345b6c4ccaa21f38c606ada79a; no voters:
I20260808 04:37:50.093235 4648 raft_consensus.cc:2781] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:37:50.093431 4648 raft_consensus.cc:686] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 LEADER]: Becoming Leader. State: Replica: 3e1fc50de4bb4098b2b77d3976fbcf03, State: Running, Role: LEADER
I20260808 04:37:50.093788 4648 consensus_queue.cc:237] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:50.097045 3968 catalog_manager.cc:5495] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e1fc50de4bb4098b2b77d3976fbcf03 (127.15.228.65). New cstate: current_term: 1 leader_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } health_report { overall_health: HEALTHY } } }
I20260808 04:37:50.122285 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:50.118286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42994 (local address 127.15.228.65:40511)
0808 04:37:50.118618 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:50.118625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:50.118638 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:37:50.118715 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:50.118718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:50.118774 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:50.118927 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:37:50.119219 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:50.119793 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:50.120659 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:50.120894 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:50.121032 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:50.121323 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:37:50.121766 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:37:50.121896 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:50.121965 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 04:37:50.122020 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":119,"threads_started":1}
I20260808 04:37:50.126585 4566 raft_consensus.cc:1261] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea [term 1 FOLLOWER]: Refusing update from remote peer 3e1fc50de4bb4098b2b77d3976fbcf03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:37:50.127050 4374 raft_consensus.cc:1261] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a [term 1 FOLLOWER]: Refusing update from remote peer 3e1fc50de4bb4098b2b77d3976fbcf03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:37:50.127769 4648 consensus_queue.cc:1035] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:37:50.128073 4657 consensus_queue.cc:1035] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:37:50.139025 4665 mvcc.cc:205] Tried to move back new op lower bound from 7316127212031569920 to 7316127211906850816. Current Snapshot: MvccSnapshot[applied={T|T < 7316127212031569920}]
I20260808 04:37:50.142160 4666 mvcc.cc:205] Tried to move back new op lower bound from 7316127212031569920 to 7316127211906850816. Current Snapshot: MvccSnapshot[applied={T|T < 7316127212031569920}]
I20260808 04:37:50.148262 3969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:37:50.148495 3969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:37:50.152261 3969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa: Sending DeleteTablet for 3 replicas of tablet 668a4f57693e4d9d905a4aa6c5cdd38f
I20260808 04:37:50.153111 4143 tablet_service.cc:1514] Processing DeleteTablet for tablet 668a4f57693e4d9d905a4aa6c5cdd38f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:37:50 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:42980
I20260808 04:37:50.153211 4351 tablet_service.cc:1514] Processing DeleteTablet for tablet 668a4f57693e4d9d905a4aa6c5cdd38f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:37:50 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:39836
I20260808 04:37:50.153455 4539 tablet_service.cc:1514] Processing DeleteTablet for tablet 668a4f57693e4d9d905a4aa6c5cdd38f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:37:50 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:60274
I20260808 04:37:50.153587 4676 tablet_replica.cc:330] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03: stopping tablet replica
I20260808 04:37:50.153720 4676 raft_consensus.cc:2229] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:37:50.154063 4676 raft_consensus.cc:2258] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:37:50.154240 4677 tablet_replica.cc:330] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a: stopping tablet replica
I20260808 04:37:50.154385 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 4023
I20260808 04:37:50.154703 4677 raft_consensus.cc:2229] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:37:50.155099 4677 raft_consensus.cc:2258] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:37:50.156828 4678 tablet_replica.cc:330] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea: stopping tablet replica
I20260808 04:37:50.157189 4678 raft_consensus.cc:2229] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:37:50.157498 4678 raft_consensus.cc:2258] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:37:50.163194 4678 ts_tablet_manager.cc:1905] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:37:50.168336 4678 ts_tablet_manager.cc:1918] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:37:50.168488 4678 log.cc:1196] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/668a4f57693e4d9d905a4aa6c5cdd38f
I20260808 04:37:50.168983 4678 ts_tablet_manager.cc:1939] T 668a4f57693e4d9d905a4aa6c5cdd38f P bedaa4b0188e4673a4cefc4dca0d48ea: Deleting consensus metadata
I20260808 04:37:50.175688 3948 catalog_manager.cc:4841] TS bedaa4b0188e4673a4cefc4dca0d48ea (127.15.228.67:34349): tablet 668a4f57693e4d9d905a4aa6c5cdd38f (table test-table [id=b7921be964d74b93ae04882392746949]) successfully deleted
I20260808 04:37:50.176609 4677 ts_tablet_manager.cc:1905] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:37:50.178915 4677 ts_tablet_manager.cc:1918] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:37:50.179118 4677 log.cc:1196] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/668a4f57693e4d9d905a4aa6c5cdd38f
I20260808 04:37:50.179425 4677 ts_tablet_manager.cc:1939] T 668a4f57693e4d9d905a4aa6c5cdd38f P c5fcc7345b6c4ccaa21f38c606ada79a: Deleting consensus metadata
I20260808 04:37:50.180691 3948 catalog_manager.cc:4841] TS c5fcc7345b6c4ccaa21f38c606ada79a (127.15.228.66:33473): tablet 668a4f57693e4d9d905a4aa6c5cdd38f (table test-table [id=b7921be964d74b93ae04882392746949]) successfully deleted
W20260808 04:37:50.182493 3949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.228.65:40511 (error 108)
I20260808 04:37:50.183216 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 4252
I20260808 04:37:50.215283 4255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:50.214747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:40511 (local address 127.0.0.1:43006)
0808 04:37:50.215026 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:50.215149 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.65:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260808 04:37:50.215588 3949 catalog_manager.cc:4584] TS 3e1fc50de4bb4098b2b77d3976fbcf03 (127.15.228.65:40511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 668a4f57693e4d9d905a4aa6c5cdd38f: Network error: Client connection negotiation failed: client connection to 127.15.228.65:40511: connect: Connection refused (error 111)
I20260808 04:37:50.221727 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 4445
I20260808 04:37:50.241532 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 3922
I20260808 04:37:50.262748 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:34873
--webserver_interface=127.15.228.126
--webserver_port=32915
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:34873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:37:50.394881 4680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:50.395200 4680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:50.395249 4680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:50.399751 4680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:37:50.399822 4680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:50.399845 4680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:50.399922 4680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:37:50.399960 4680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:37:50.406103 4680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:34873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:34873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=32915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:50.407286 4680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:50.408490 4680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:50.419863 4697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:50.419788 4686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:50.419788 4694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:50.420279 4680 server_base.cc:1034] running on GCE node
I20260808 04:37:50.420912 4680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:50.422576 4680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:50.423888 4680 hybrid_clock.cc:648] HybridClock initialized: now 1786163870423874 us; error 43 us; skew 500 ppm
Aug 08 04:37:50 dist-test-slave-drl0 krb5kdc[3152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163870, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:50.427811 4680 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:37:50.429150 4680 webserver.cc:460] Webserver started at http://127.15.228.126:32915/ using document root <none> and password file <none>
I20260808 04:37:50.429993 4680 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:50.430138 4680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:37:50.434379 4680 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:37:50.436913 4709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:50.438405 4680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:37:50.438665 4680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "53d25bfe01ec48abb6226f2b1aef9afa"
format_stamp: "Formatted at 2026-08-08 04:37:47 on dist-test-slave-drl0"
server_key: "bf07bf9316c10744bf43fb6ee7a8cd26"
server_key_iv: "40fa369efbf0a463793da8f15502ba74"
server_key_version: "encryptionkey@0"
I20260808 04:37:50.439383 4680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:50.454459 4680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:50.458036 4680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:50.458278 4680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:50.467973 4680 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:34873
I20260808 04:37:50.468501 4770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:34873 every 8 connection(s)
I20260808 04:37:50.469323 4680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:37:50.471612 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 4680
I20260808 04:37:50.472114 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:40511
--local_ip_for_outbound_sockets=127.15.228.65
--tserver_master_addrs=127.15.228.126:34873
--webserver_port=42241
--webserver_interface=127.15.228.65
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:37:50.474108 4772 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:37:50.475900 4772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa: Bootstrap starting.
I20260808 04:37:50.496160 4772 log.cc:824] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa: Log is configured to *not* fsync() on all Append() calls
I20260808 04:37:50.508414 4772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:37:50.508977 4772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa: Bootstrap complete.
I20260808 04:37:50.514609 4772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34873 } }
I20260808 04:37:50.515146 4772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d25bfe01ec48abb6226f2b1aef9afa, State: Initialized, Role: FOLLOWER
I20260808 04:37:50.515678 4772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34873 } }
I20260808 04:37:50.515834 4772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:37:50.515934 4772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:37:50.516057 4772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 2 FOLLOWER]: Advancing to term 3
I20260808 04:37:50.518342 4772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34873 } }
I20260808 04:37:50.518811 4772 leader_election.cc:302] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53d25bfe01ec48abb6226f2b1aef9afa; no voters:
I20260808 04:37:50.519218 4772 leader_election.cc:288] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 04:37:50.519467 4778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 3 FOLLOWER]: Leader election won for term 3
I20260808 04:37:50.519932 4778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [term 3 LEADER]: Becoming Leader. State: Replica: 53d25bfe01ec48abb6226f2b1aef9afa, State: Running, Role: LEADER
I20260808 04:37:50.520309 4778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34873 } }
I20260808 04:37:50.520607 4772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:37:50.521986 4780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53d25bfe01ec48abb6226f2b1aef9afa. Latest consensus state: current_term: 3 leader_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34873 } } }
I20260808 04:37:50.522045 4779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d25bfe01ec48abb6226f2b1aef9afa" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34873 } } }
I20260808 04:37:50.522159 4779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [sys.catalog]: This master's current role is: LEADER
I20260808 04:37:50.522107 4780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa [sys.catalog]: This master's current role is: LEADER
I20260808 04:37:50.526503 4802 catalog_manager.cc:1259] Loaded cluster ID: 273838deed8842cdb8d537d1ee70de99
I20260808 04:37:50.526579 4802 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa: loading cluster ID for follower catalog manager: success
W20260808 04:37:50.528163 4802 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 53d25bfe01ec48abb6226f2b1aef9afa: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260808 04:37:50.528260 4802 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260808 04:37:50.528352 4790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:37:50.530869 4790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:37:50.531458 4790 catalog_manager.cc:1259] Loaded cluster ID: 273838deed8842cdb8d537d1ee70de99
I20260808 04:37:50.531531 4790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 04:37:50.532234 4790 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6076d8f9ad google::LogMessage::Fail() at ??:0
@ 0x7f6076d9288a google::LogMessage::SendToLog() at ??:0
@ 0x7f6076d8f4de google::LogMessage::Flush() at ??:0
@ 0x7f6076d90a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f60816cefdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f60816cfd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f60816cc579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f6081710640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6078fe13d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6078fe1c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f6078fe33f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6078fd2512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f607a6166db start_thread at ??:0
@ 0x7f607588d61f clone at ??:0
W20260808 04:37:50.648640 4774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:50.648998 4774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:50.649060 4774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:50.655478 4774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:50.655601 4774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:50.655704 4774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:37:50.664723 4774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:40511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=42241
--tserver_master_addrs=127.15.228.126:34873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:50.666595 4774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:50.668650 4774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:50.683099 4843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:50.683157 4844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:50.682760 4847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:50.683799 4774 server_base.cc:1034] running on GCE node
I20260808 04:37:50.684347 4774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:50.685676 4774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:50.686918 4774 hybrid_clock.cc:648] HybridClock initialized: now 1786163870686892 us; error 44 us; skew 500 ppm
Aug 08 04:37:50 dist-test-slave-drl0 krb5kdc[3152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163870, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:50.690707 4774 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:37:50.692363 4774 webserver.cc:460] Webserver started at http://127.15.228.65:42241/ using document root <none> and password file <none>
I20260808 04:37:50.693451 4774 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:50.693712 4774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:37:50.699307 4774 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:37:50.702435 4861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:50.704699 4774 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:37:50.704883 4774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "3e1fc50de4bb4098b2b77d3976fbcf03"
format_stamp: "Formatted at 2026-08-08 04:37:47 on dist-test-slave-drl0"
server_key: "34b67d3a73530ee9e7342b88cc1968e0"
server_key_iv: "43f533ec3d898e5673d0ce952e642c74"
server_key_version: "encryptionkey@0"
I20260808 04:37:50.705643 4774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:50.738121 4774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:50.742159 4774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:50.742398 4774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:50.743117 4774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:37:50.744364 4869 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 04:37:50.750903 4774 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 04:37:50.751081 4774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.001s
I20260808 04:37:50.751174 4774 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 04:37:50.752933 4774 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 04:37:50.753018 4774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260808 04:37:50.753319 4869 tablet_bootstrap.cc:492] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03: Bootstrap starting.
I20260808 04:37:50.765380 4774 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:40511
I20260808 04:37:50.765820 5017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:40511 every 8 connection(s)
I20260808 04:37:50.767032 4774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:37:50.776273 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 4774
I20260808 04:37:50.776973 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:33473
--local_ip_for_outbound_sockets=127.15.228.66
--tserver_master_addrs=127.15.228.126:34873
--webserver_port=42265
--webserver_interface=127.15.228.66
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:37:50.782794 4869 log.cc:824] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03: Log is configured to *not* fsync() on all Append() calls
W20260808 04:37:50.787021 5021 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:37:50.769898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:33709)
0808 04:37:50.770630 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:50.770699 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:50.772168 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:50.785752 (+ 13584us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":226,"threads_started":1}
W20260808 04:37:50.787904 5018 heartbeater.cc:644] Failed to heartbeat to 127.15.228.126:34873 (0 consecutive failures): Network error: Failed to ping master at 127.15.228.126:34873: Client connection negotiation failed: client connection to 127.15.228.126:34873: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 04:37:50.788837 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:50.788447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:46013)
0808 04:37:50.788628 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:50.788732 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 04:37:50.789714 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:50.789425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:59859)
0808 04:37:50.789569 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:50.789631 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 04:37:50.789980 5018 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 04:37:50.792371 4869 tablet_bootstrap.cc:492] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:37:50.792936 4869 tablet_bootstrap.cc:492] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03: Bootstrap complete.
I20260808 04:37:50.794590 4869 ts_tablet_manager.cc:1397] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03: Time spent bootstrapping tablet: real 0.042s user 0.016s sys 0.015s
I20260808 04:37:50.797420 4869 raft_consensus.cc:348] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:50.797886 4869 raft_consensus.cc:729] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e1fc50de4bb4098b2b77d3976fbcf03, State: Initialized, Role: FOLLOWER
I20260808 04:37:50.798362 4869 consensus_queue.cc:260] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:50.799595 4869 ts_tablet_manager.cc:1428] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
I20260808 04:37:50.799685 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:50.799380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:38075)
0808 04:37:50.799535 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:50.799604 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 04:37:50.800431 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:50.800176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:47145)
0808 04:37:50.800314 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:50.800377 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260808 04:37:50.914970 5022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:50.915256 5022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:50.915308 5022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:50.919464 5022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:50.919529 5022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:50.919589 5022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:37:50.925177 5022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:33473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=42265
--tserver_master_addrs=127.15.228.126:34873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:50.926358 5022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:50.927527 5022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:50.939924 5034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:50.940196 5022 server_base.cc:1034] running on GCE node
W20260808 04:37:50.939975 5031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:50.939946 5037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:50.940984 5022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:50.941642 5022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:50.942843 5022 hybrid_clock.cc:648] HybridClock initialized: now 1786163870942827 us; error 33 us; skew 500 ppm
Aug 08 04:37:50 dist-test-slave-drl0 krb5kdc[3152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163870, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:50.946614 5022 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:37:50.947659 5022 webserver.cc:460] Webserver started at http://127.15.228.66:42265/ using document root <none> and password file <none>
I20260808 04:37:50.948287 5022 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:50.948357 5022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:37:50.953233 5022 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:37:50.955698 5044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:50.957130 5022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:37:50.957284 5022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "c5fcc7345b6c4ccaa21f38c606ada79a"
format_stamp: "Formatted at 2026-08-08 04:37:48 on dist-test-slave-drl0"
server_key: "c8f9044eb78ee1829f7cd8d933e611d8"
server_key_iv: "eb154396f3a905ba580482b27ec47501"
server_key_version: "encryptionkey@0"
I20260808 04:37:50.957859 5022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:50.987139 5022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:50.990916 5022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:50.991163 5022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:50.991875 5022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:37:50.992856 5022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:37:50.992931 5022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:50.993005 5022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:37:50.993058 5022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:51.003400 5022 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:33473
I20260808 04:37:51.003437 5205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:33473 every 8 connection(s)
I20260808 04:37:51.004457 5022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:37:51.010104 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 5022
I20260808 04:37:51.010581 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:51.007951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.66:34863)
0808 04:37:51.008751 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:51.009353 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":212,"threads_started":1}
I20260808 04:37:51.010677 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:34349
--local_ip_for_outbound_sockets=127.15.228.67
--tserver_master_addrs=127.15.228.126:34873
--webserver_port=39225
--webserver_interface=127.15.228.67
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:37:51.011440 5206 heartbeater.cc:644] Failed to heartbeat to 127.15.228.126:34873 (0 consecutive failures): Network error: Failed to ping master at 127.15.228.126:34873: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
I20260808 04:37:51.012477 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:51.012114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.66:37321)
0808 04:37:51.012309 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:51.012382 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260808 04:37:51.013459 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:51.013023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.66:33397)
0808 04:37:51.013229 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:51.013338 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260808 04:37:51.013828 5206 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:37:51.147516 5211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:37:51.147814 5211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:37:51.147908 5211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:37:51.151930 5211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:37:51.152019 5211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:37:51.152107 5211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:37:51.157449 5211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:34349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=39225
--tserver_master_addrs=127.15.228.126:34873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:37:51.158604 5211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:37:51.159736 5211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:37:51.172569 5221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:51.172529 5220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:37:51.172523 5224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:37:51.172816 5211 server_base.cc:1034] running on GCE node
I20260808 04:37:51.173285 5211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:37:51.174114 5211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:37:51.175316 5211 hybrid_clock.cc:648] HybridClock initialized: now 1786163871175282 us; error 48 us; skew 500 ppm
Aug 08 04:37:51 dist-test-slave-drl0 krb5kdc[3152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163871, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:37:51.181385 5211 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:37:51.182667 5211 webserver.cc:460] Webserver started at http://127.15.228.67:39225/ using document root <none> and password file <none>
I20260808 04:37:51.183516 5211 fs_manager.cc:359] Metadata directory not provided
I20260808 04:37:51.183622 5211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:37:51.189105 5211 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260808 04:37:51.191816 5235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:51.193979 5211 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:37:51.194123 5211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "bedaa4b0188e4673a4cefc4dca0d48ea"
format_stamp: "Formatted at 2026-08-08 04:37:48 on dist-test-slave-drl0"
server_key: "73b4dcaa4656c17d741931a8ee75af6e"
server_key_iv: "aebd2605e82b9df8ba210b22dabb2001"
server_key_version: "encryptionkey@0"
I20260808 04:37:51.194917 5211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:37:51.240268 5211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:37:51.244637 5211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:37:51.244813 5211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:37:51.245609 5211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:37:51.246469 5211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:37:51.246521 5211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:51.246562 5211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:37:51.246578 5211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:37:51.259651 5211 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:34349
I20260808 04:37:51.259755 5394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:34349 every 8 connection(s)
I20260808 04:37:51.260810 5211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:37:51.261894 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 5211
I20260808 04:37:51.265585 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:51.263720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.67:35345)
0808 04:37:51.264224 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:51.264335 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":123,"threads_started":1}
W20260808 04:37:51.266268 5395 heartbeater.cc:644] Failed to heartbeat to 127.15.228.126:34873 (0 consecutive failures): Network error: Failed to ping master at 127.15.228.126:34873: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
I20260808 04:37:51.267612 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:51.266754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.67:40039)
0808 04:37:51.267347 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:51.267450 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410}
I20260808 04:37:51.268757 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:51.268456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.67:55439)
0808 04:37:51.268632 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:51.268694 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260808 04:37:51.269091 5395 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 04:37:51.801970 5402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:51.801359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:58149)
0808 04:37:51.801704 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:51.801857 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":83,"threads_started":1}
I20260808 04:37:52.015300 5403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:52.014765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.66:33027)
0808 04:37:52.015119 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:52.015182 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":83,"threads_started":1}
I20260808 04:37:52.270623 5406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:52.270075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.67:37823)
0808 04:37:52.270444 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:52.270509 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":106,"threads_started":1}
I20260808 04:37:52.625262 5407 raft_consensus.cc:482] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:37:52.625607 5407 raft_consensus.cc:504] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:52.627827 5407 leader_election.cc:288] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5fcc7345b6c4ccaa21f38c606ada79a (127.15.228.66:33473), bedaa4b0188e4673a4cefc4dca0d48ea (127.15.228.67:34349)
Aug 08 04:37:52 dist-test-slave-drl0 krb5kdc[3152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163870, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.67@KRBTEST.COM
Aug 08 04:37:52 dist-test-slave-drl0 krb5kdc[3152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163870, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.66@KRBTEST.COM
I20260808 04:37:52.643061 5409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:52.628623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:48659 (local address 127.15.228.66:33473)
0808 04:37:52.629044 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:52.629061 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:52.629226 (+ 165us) server_negotiation.cc:407] Connection header received
0808 04:37:52.630173 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:52.630180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:52.630498 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:52.630832 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:52.632209 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:52.633182 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:52.634454 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:52.634919 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:52.635140 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:52.638480 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:52.638485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:52.638495 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:52.638523 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:52.640320 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:52.640915 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:52.640923 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:52.640931 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:52.641065 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:52.641462 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:52.641465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:52.641466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:52.641862 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:52.642006 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:52.642562 (+ 556us) server_negotiation.cc:299] Negotiation successful
0808 04:37:52.642837 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":154,"threads_started":1}
I20260808 04:37:52.643049 5408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:52.628824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:34349 (local address 127.15.228.65:53737)
0808 04:37:52.629112 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:52.629126 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:52.629247 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:52.630879 (+ 1632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:52.630885 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:52.631517 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:52.632023 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:52.632040 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:52.633325 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:52.633327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:52.634302 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:52.634313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:52.635072 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:52.635076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:52.635326 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:52.636081 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:52.636101 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:52.637844 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:52.640372 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:52.640381 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:52.640399 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:52.640803 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:52.641208 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:52.641210 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:52.641212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:52.641327 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:52.641978 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:52.641984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:52.642246 (+ 262us) client_negotiation.cc:769] Sending connection context
0808 04:37:52.642427 (+ 181us) client_negotiation.cc:240] Negotiation successful
0808 04:37:52.642776 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":203,"spinlock_wait_cycles":73600,"thread_start_us":70,"threads_started":1}
I20260808 04:37:52.643066 5410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:52.628382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:33473 (local address 127.15.228.65:48659)
0808 04:37:52.629086 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:52.629101 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:37:52.629247 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:37:52.630880 (+ 1633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:37:52.630886 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:37:52.631517 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:37:52.632024 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:52.632039 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:52.633325 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:52.633329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:52.634288 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:37:52.634299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:37:52.635101 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:37:52.635104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:37:52.635288 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:52.636081 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:37:52.636104 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:37:52.638190 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:37:52.640455 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:52.640457 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:52.640459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:52.640795 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:52.641170 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:37:52.641173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:37:52.641175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:37:52.641333 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:37:52.641995 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:37:52.641998 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:37:52.642238 (+ 240us) client_negotiation.cc:769] Sending connection context
0808 04:37:52.642427 (+ 189us) client_negotiation.cc:240] Negotiation successful
0808 04:37:52.642866 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":334,"spinlock_wait_cycles":45440,"thread_start_us":121,"threads_started":1}
I20260808 04:37:52.643075 5411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:52.629099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:53737 (local address 127.15.228.67:34349)
0808 04:37:52.629438 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0808 04:37:52.629467 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0808 04:37:52.629493 (+ 26us) server_negotiation.cc:407] Connection header received
0808 04:37:52.630147 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:37:52.630152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:37:52.630498 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:37:52.630832 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:37:52.632198 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:52.633167 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:52.634428 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:37:52.634891 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:37:52.635115 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:37:52.638102 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:37:52.638110 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:37:52.638128 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:37:52.638169 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:37:52.640237 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:52.640961 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:52.640966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:52.640970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:52.641080 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:37:52.641431 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:37:52.641438 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:37:52.641441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:37:52.641855 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:37:52.641995 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 04:37:52.642558 (+ 563us) server_negotiation.cc:299] Negotiation successful
0808 04:37:52.642858 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":158,"threads_started":1}
W20260808 04:37:52.646013 4862 leader_election.cc:341] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c5fcc7345b6c4ccaa21f38c606ada79a (127.15.228.66:33473): Not found: Tablet not found: 668a4f57693e4d9d905a4aa6c5cdd38f
W20260808 04:37:52.646214 4862 leader_election.cc:341] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bedaa4b0188e4673a4cefc4dca0d48ea (127.15.228.67:34349): Not found: Tablet not found: 668a4f57693e4d9d905a4aa6c5cdd38f
I20260808 04:37:52.646358 4862 leader_election.cc:302] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e1fc50de4bb4098b2b77d3976fbcf03; no voters: bedaa4b0188e4673a4cefc4dca0d48ea, c5fcc7345b6c4ccaa21f38c606ada79a
I20260808 04:37:52.646701 5407 raft_consensus.cc:2726] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:37:52.803690 5410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:52.803343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:44205)
0808 04:37:52.803516 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:52.803591 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260808 04:37:53.016942 5415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:53.016357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.66:57443)
0808 04:37:53.016668 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:53.016727 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":93,"threads_started":1}
I20260808 04:37:53.272521 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:53.271905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.67:47921)
0808 04:37:53.272288 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:53.272359 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":122,"threads_started":1}
I20260808 04:37:53.805322 5418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:53.804794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:45317)
0808 04:37:53.805155 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:53.805215 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":92,"threads_started":1}
I20260808 04:37:54.018327 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:54.017887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.66:32955)
0808 04:37:54.018157 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:54.018214 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
I20260808 04:37:54.274286 5420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:54.273490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.67:44209)
0808 04:37:54.273944 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:54.274046 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":83,"threads_started":1}
I20260808 04:37:54.695202 5421 raft_consensus.cc:482] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:37:54.695370 5421 raft_consensus.cc:504] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:54.695910 5421 leader_election.cc:288] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5fcc7345b6c4ccaa21f38c606ada79a (127.15.228.66:33473), bedaa4b0188e4673a4cefc4dca0d48ea (127.15.228.67:34349)
W20260808 04:37:54.697212 4862 leader_election.cc:341] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c5fcc7345b6c4ccaa21f38c606ada79a (127.15.228.66:33473): Not found: Tablet not found: 668a4f57693e4d9d905a4aa6c5cdd38f
W20260808 04:37:54.697348 4862 leader_election.cc:341] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bedaa4b0188e4673a4cefc4dca0d48ea (127.15.228.67:34349): Not found: Tablet not found: 668a4f57693e4d9d905a4aa6c5cdd38f
I20260808 04:37:54.697412 4862 leader_election.cc:302] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e1fc50de4bb4098b2b77d3976fbcf03; no voters: bedaa4b0188e4673a4cefc4dca0d48ea, c5fcc7345b6c4ccaa21f38c606ada79a
I20260808 04:37:54.697747 5421 raft_consensus.cc:2726] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:37:54.806633 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:54.806175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:38325)
0808 04:37:54.806466 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:54.806530 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":81,"threads_started":1}
I20260808 04:37:55.020174 5423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:55.019611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.66:33331)
0808 04:37:55.019984 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:55.020062 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":117,"threads_started":1}
I20260808 04:37:55.275779 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:55.275323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.67:48741)
0808 04:37:55.275614 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:55.275671 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":81,"threads_started":1}
I20260808 04:37:55.808496 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:55.807922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:50261)
0808 04:37:55.808315 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:55.808384 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":154,"threads_started":1}
I20260808 04:37:56.021724 5426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:56.021240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.66:58025)
0808 04:37:56.021557 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:56.021621 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":94,"threads_started":1}
I20260808 04:37:56.277344 5427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:56.276871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.67:37755)
0808 04:37:56.277178 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:56.277249 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":94,"threads_started":1}
I20260808 04:37:56.429955 5428 raft_consensus.cc:482] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:37:56.430126 5428 raft_consensus.cc:504] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:56.430650 5428 leader_election.cc:288] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5fcc7345b6c4ccaa21f38c606ada79a (127.15.228.66:33473), bedaa4b0188e4673a4cefc4dca0d48ea (127.15.228.67:34349)
W20260808 04:37:56.431818 4862 leader_election.cc:341] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c5fcc7345b6c4ccaa21f38c606ada79a (127.15.228.66:33473): Not found: Tablet not found: 668a4f57693e4d9d905a4aa6c5cdd38f
W20260808 04:37:56.432112 4862 leader_election.cc:341] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bedaa4b0188e4673a4cefc4dca0d48ea (127.15.228.67:34349): Not found: Tablet not found: 668a4f57693e4d9d905a4aa6c5cdd38f
I20260808 04:37:56.432197 4862 leader_election.cc:302] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e1fc50de4bb4098b2b77d3976fbcf03; no voters: bedaa4b0188e4673a4cefc4dca0d48ea, c5fcc7345b6c4ccaa21f38c606ada79a
I20260808 04:37:56.432433 5428 raft_consensus.cc:2726] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:37:56.809962 5429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:56.809425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:37707)
0808 04:37:56.809718 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:56.809806 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":117,"threads_started":1}
I20260808 04:37:57.023423 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:57.022916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.66:38059)
0808 04:37:57.023260 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:57.023314 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":77,"threads_started":1}
I20260808 04:37:57.279397 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:57.278598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.67:42591)
0808 04:37:57.279141 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:57.279240 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":162,"threads_started":1}
I20260808 04:37:57.811355 5432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:57.810756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:35613)
0808 04:37:57.811129 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:57.811212 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":81,"threads_started":1}
I20260808 04:37:58.024852 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:58.024332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.66:49331)
0808 04:37:58.024696 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:58.024752 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":103,"threads_started":1}
I20260808 04:37:58.265451 5435 raft_consensus.cc:482] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:37:58.265643 5435 raft_consensus.cc:504] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:37:58.266304 5435 leader_election.cc:288] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5fcc7345b6c4ccaa21f38c606ada79a (127.15.228.66:33473), bedaa4b0188e4673a4cefc4dca0d48ea (127.15.228.67:34349)
W20260808 04:37:58.267378 4862 leader_election.cc:341] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c5fcc7345b6c4ccaa21f38c606ada79a (127.15.228.66:33473): Not found: Tablet not found: 668a4f57693e4d9d905a4aa6c5cdd38f
W20260808 04:37:58.267664 4862 leader_election.cc:341] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bedaa4b0188e4673a4cefc4dca0d48ea (127.15.228.67:34349): Not found: Tablet not found: 668a4f57693e4d9d905a4aa6c5cdd38f
I20260808 04:37:58.267751 4862 leader_election.cc:302] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e1fc50de4bb4098b2b77d3976fbcf03; no voters: bedaa4b0188e4673a4cefc4dca0d48ea, c5fcc7345b6c4ccaa21f38c606ada79a
I20260808 04:37:58.267951 5435 raft_consensus.cc:2726] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:37:58.281333 5437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:58.280491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.67:43057)
0808 04:37:58.281060 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:58.281196 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":106,"threads_started":1}
I20260808 04:37:58.812690 5438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:58.812225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:48429)
0808 04:37:58.812518 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:58.812577 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":85,"threads_started":1}
I20260808 04:37:59.026383 5439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:59.025887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.66:49863)
0808 04:37:59.026217 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:59.026279 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":98,"threads_started":1}
I20260808 04:37:59.283217 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:59.282464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.67:58591)
0808 04:37:59.282840 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:59.283058 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":80,"threads_started":1}
I20260808 04:37:59.814607 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:37:59.814083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:42595)
0808 04:37:59.814429 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0808 04:37:59.814498 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":97,"threads_started":1}
I20260808 04:38:00.028560 5442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:00.027787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.66:54013)
0808 04:38:00.028310 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:00.028410 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":171,"threads_started":1}
I20260808 04:38:00.285094 5444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:00.284406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.67:57645)
0808 04:38:00.284918 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:00.284983 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":97,"threads_started":1}
I20260808 04:38:00.816428 5445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:00.815904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:57185)
0808 04:38:00.816245 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:00.816314 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":86,"threads_started":1}
I20260808 04:38:01.030217 5446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:01.029745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.66:48845)
0808 04:38:01.030047 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:01.030112 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":84,"threads_started":1}
I20260808 04:38:01.287112 5447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:01.286483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.67:45145)
0808 04:38:01.286857 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:01.286988 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":112,"threads_started":1}
I20260808 04:38:01.593658 5448 raft_consensus.cc:482] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:38:01.593865 5448 raft_consensus.cc:504] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:38:01.594470 5448 leader_election.cc:288] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5fcc7345b6c4ccaa21f38c606ada79a (127.15.228.66:33473), bedaa4b0188e4673a4cefc4dca0d48ea (127.15.228.67:34349)
W20260808 04:38:01.595830 4862 leader_election.cc:341] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c5fcc7345b6c4ccaa21f38c606ada79a (127.15.228.66:33473): Not found: Tablet not found: 668a4f57693e4d9d905a4aa6c5cdd38f
W20260808 04:38:01.596073 4862 leader_election.cc:341] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bedaa4b0188e4673a4cefc4dca0d48ea (127.15.228.67:34349): Not found: Tablet not found: 668a4f57693e4d9d905a4aa6c5cdd38f
I20260808 04:38:01.596153 4862 leader_election.cc:302] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e1fc50de4bb4098b2b77d3976fbcf03; no voters: bedaa4b0188e4673a4cefc4dca0d48ea, c5fcc7345b6c4ccaa21f38c606ada79a
I20260808 04:38:01.596380 5448 raft_consensus.cc:2726] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:38:01.817832 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:01.817356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:41111)
0808 04:38:01.817663 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:01.817726 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":84,"threads_started":1}
I20260808 04:38:02.032069 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:02.031386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.66:41329)
0808 04:38:02.031886 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:02.031964 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":140,"threads_started":1}
I20260808 04:38:02.289839 5452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:02.289160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.67:34023)
0808 04:38:02.289600 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:02.289680 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":157,"threads_started":1}
I20260808 04:38:02.819213 5453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:02.818694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:42613)
0808 04:38:02.819058 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:02.819114 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":97,"threads_started":1}
I20260808 04:38:03.033856 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:03.033208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.66:42823)
0808 04:38:03.033648 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:03.033743 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":141,"threads_started":1}
I20260808 04:38:03.291232 5457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:03.290676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.67:34425)
0808 04:38:03.291022 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:03.291108 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":82,"threads_started":1}
I20260808 04:38:03.820571 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:03.820098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:54439)
0808 04:38:03.820404 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:03.820466 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":86,"threads_started":1}
I20260808 04:38:04.035542 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:04.035091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.66:59417)
0808 04:38:04.035378 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:04.035437 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
I20260808 04:38:04.293414 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:04.292665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.67:43113)
0808 04:38:04.293191 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:04.293273 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":116,"threads_started":1}
I20260808 04:38:04.822156 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:04.821661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:52045)
0808 04:38:04.821978 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:04.822051 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":90,"threads_started":1}
I20260808 04:38:05.037060 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:05.036569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.66:51643)
0808 04:38:05.036895 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:05.036959 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":94,"threads_started":1}
I20260808 04:38:05.295300 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:05.294656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.67:42277)
0808 04:38:05.295128 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:05.295191 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":146,"threads_started":1}
I20260808 04:38:05.339843 5468 raft_consensus.cc:482] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:38:05.340035 5468 raft_consensus.cc:504] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fcc7345b6c4ccaa21f38c606ada79a" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 33473 } } peers { permanent_uuid: "bedaa4b0188e4673a4cefc4dca0d48ea" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 34349 } } peers { permanent_uuid: "3e1fc50de4bb4098b2b77d3976fbcf03" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 40511 } }
I20260808 04:38:05.340550 5468 leader_election.cc:288] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5fcc7345b6c4ccaa21f38c606ada79a (127.15.228.66:33473), bedaa4b0188e4673a4cefc4dca0d48ea (127.15.228.67:34349)
W20260808 04:38:05.341681 4862 leader_election.cc:341] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c5fcc7345b6c4ccaa21f38c606ada79a (127.15.228.66:33473): Not found: Tablet not found: 668a4f57693e4d9d905a4aa6c5cdd38f
W20260808 04:38:05.341790 4862 leader_election.cc:341] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bedaa4b0188e4673a4cefc4dca0d48ea (127.15.228.67:34349): Not found: Tablet not found: 668a4f57693e4d9d905a4aa6c5cdd38f
I20260808 04:38:05.341961 4862 leader_election.cc:302] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e1fc50de4bb4098b2b77d3976fbcf03; no voters: bedaa4b0188e4673a4cefc4dca0d48ea, c5fcc7345b6c4ccaa21f38c606ada79a
I20260808 04:38:05.342224 5468 raft_consensus.cc:2726] T 668a4f57693e4d9d905a4aa6c5cdd38f P 3e1fc50de4bb4098b2b77d3976fbcf03 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:38:05.823959 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:05.823491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.65:42771)
0808 04:38:05.823791 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:05.823858 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":81,"threads_started":1}
I20260808 04:38:06.039053 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:06.038450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.66:44313)
0808 04:38:06.038802 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:06.038870 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":89,"threads_started":1}
I20260808 04:38:06.275240 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 4774
I20260808 04:38:06.289476 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 5022
I20260808 04:38:06.297588 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:06.297151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:34873 (local address 127.15.228.67:51835)
0808 04:38:06.297437 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:06.297495 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:34873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
I20260808 04:38:06.303539 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 5211
2026-08-08T04:38:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (20563 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:38:06 dist-test-slave-drl0 krb5kdc[5477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:38:06 dist-test-slave-drl0 krb5kdc[5477](info): set up 2 sockets
Aug 08 04:38:06 dist-test-slave-drl0 krb5kdc[5477](info): commencing operation
krb5kdc: starting...
W20260808 04:38:07.915786 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:38:08 dist-test-slave-drl0 krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:38:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:38:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:38:08.059129 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:45107
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:45107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:38:08.187945 5491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:38:08.188213 5491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:38:08.188256 5491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:38:08.192762 5491 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 04:38:08.192831 5491 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 04:38:08.192849 5491 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 04:38:08.192864 5491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:38:08.192883 5491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:38:08.192936 5491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:38:08.192981 5491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:38:08.193003 5491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:38:08.198864 5491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:45107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:45107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:38:08.200042 5491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:38:08.201136 5491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:38:08.210706 5502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:38:08.210755 5499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:38:08.211071 5497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:38:08.210862 5491 server_base.cc:1034] running on GCE node
I20260808 04:38:08.212023 5491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:38:08.213220 5491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:38:08.214540 5491 hybrid_clock.cc:648] HybridClock initialized: now 1786163888214458 us; error 92 us; skew 500 ppm
Aug 08 04:38:08 dist-test-slave-drl0 krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163888, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:38:08.217819 5491 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:38:08.218760 5491 webserver.cc:460] Webserver started at http://127.15.228.126:45783/ using document root <none> and password file <none>
I20260808 04:38:08.219429 5491 fs_manager.cc:359] Metadata directory not provided
I20260808 04:38:08.219506 5491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:38:08.219686 5491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:38:08.221961 5491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "5025a708d2244712a14fe71c6895fe99"
format_stamp: "Formatted at 2026-08-08 04:38:08 on dist-test-slave-drl0"
server_key: "fd46ca6520eaf2271a69f50cf2ea5979"
server_key_iv: "23ae778879d5bae8503a56739fedf955"
server_key_version: "encryptionkey@0"
I20260808 04:38:08.222504 5491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "5025a708d2244712a14fe71c6895fe99"
format_stamp: "Formatted at 2026-08-08 04:38:08 on dist-test-slave-drl0"
server_key: "fd46ca6520eaf2271a69f50cf2ea5979"
server_key_iv: "23ae778879d5bae8503a56739fedf955"
server_key_version: "encryptionkey@0"
I20260808 04:38:08.227119 5491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260808 04:38:08.230350 5516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:08.231953 5491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:38:08.232129 5491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "5025a708d2244712a14fe71c6895fe99"
format_stamp: "Formatted at 2026-08-08 04:38:08 on dist-test-slave-drl0"
server_key: "fd46ca6520eaf2271a69f50cf2ea5979"
server_key_iv: "23ae778879d5bae8503a56739fedf955"
server_key_version: "encryptionkey@0"
I20260808 04:38:08.232254 5491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:38:08.243328 5491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:38:08.246795 5491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:38:08.247066 5491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:38:08.255558 5491 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:45107
I20260808 04:38:08.256075 5593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:45107 every 8 connection(s)
I20260808 04:38:08.257092 5491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:38:08.260592 5594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:38:08.266224 5594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99: Bootstrap starting.
I20260808 04:38:08.267411 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 5491
I20260808 04:38:08.267589 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:38:08.267889 16273 external_mini_cluster.cc:1428] Setting key d76ce04f0ac0d80d3043df26d8c07353
I20260808 04:38:08.269105 5594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99: Neither blocks nor log segments found. Creating new log.
I20260808 04:38:08.269975 5594 log.cc:824] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99: Log is configured to *not* fsync() on all Append() calls
I20260808 04:38:08.272629 5594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99: No bootstrap required, opened a new log
I20260808 04:38:08.276561 5594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5025a708d2244712a14fe71c6895fe99" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45107 } }
I20260808 04:38:08.276825 5594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:38:08.276928 5594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5025a708d2244712a14fe71c6895fe99, State: Initialized, Role: FOLLOWER
I20260808 04:38:08.277480 5594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5025a708d2244712a14fe71c6895fe99" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45107 } }
I20260808 04:38:08.277653 5594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:38:08.277752 5594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:38:08.277884 5594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:38:08.278972 5594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5025a708d2244712a14fe71c6895fe99" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45107 } }
I20260808 04:38:08.279350 5594 leader_election.cc:302] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5025a708d2244712a14fe71c6895fe99; no voters:
Aug 08 04:38:08 dist-test-slave-drl0 krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:38:08.279738 5594 leader_election.cc:288] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:38:08.279845 5602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:38:08.280109 5602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 1 LEADER]: Becoming Leader. State: Replica: 5025a708d2244712a14fe71c6895fe99, State: Running, Role: LEADER
I20260808 04:38:08.280530 5602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5025a708d2244712a14fe71c6895fe99" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45107 } }
I20260808 04:38:08.281188 5594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:38:08.281812 5602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5025a708d2244712a14fe71c6895fe99. Latest consensus state: current_term: 1 leader_uuid: "5025a708d2244712a14fe71c6895fe99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5025a708d2244712a14fe71c6895fe99" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45107 } } }
I20260808 04:38:08.283526 5602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [sys.catalog]: This master's current role is: LEADER
I20260808 04:38:08.283123 5603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5025a708d2244712a14fe71c6895fe99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5025a708d2244712a14fe71c6895fe99" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45107 } } }
I20260808 04:38:08.283919 5603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [sys.catalog]: This master's current role is: LEADER
W20260808 04:38:08.287593 5621 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:38:08.287691 5621 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:38:08.287953 5613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:38:08.288878 5613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:38:08.289053 5597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:08.269094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55752 (local address 127.15.228.126:45107)
0808 04:38:08.269805 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:08.269829 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:08.269886 (+ 57us) server_negotiation.cc:407] Connection header received
0808 04:38:08.271198 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:08.271227 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:08.271698 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:08.272173 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:38:08.273247 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:08.274381 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:08.275716 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:08.276381 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:08.276617 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:08.280721 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:38:08.280733 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:38:08.280761 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:38:08.280820 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:38:08.283654 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:08.284379 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:08.284388 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:08.284397 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:08.284564 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:08.285127 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:08.285132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:08.285134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:08.286154 (+ 1020us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:38:08.286444 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:08.286850 (+ 406us) server_negotiation.cc:299] Negotiation successful
0808 04:38:08.287742 (+ 892us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":150,"threads_started":1}
I20260808 04:38:08.295644 5613 catalog_manager.cc:1347] Generated new cluster ID: 53b08da430a7494c97ae706acd573c3b
I20260808 04:38:08.295737 5613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:38:08.324050 5613 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:38:08.325296 5613 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:38:08.345278 5613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99: Generated new TSK 0
I20260808 04:38:08.346673 5613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
I20260808 04:38:08.387419 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:45107
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:38:08.508942 5628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:38:08.509198 5628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:38:08.509243 5628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:38:08.513264 5628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:38:08.513321 5628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:38:08.513379 5628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:38:08.518579 5628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:38:08.519673 5628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:38:08.520725 5628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:38:08.531769 5637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:38:08.531767 5633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:38:08.531816 5635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:38:08.532549 5628 server_base.cc:1034] running on GCE node
I20260808 04:38:08.533304 5628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:38:08.534281 5628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:38:08.535550 5628 hybrid_clock.cc:648] HybridClock initialized: now 1786163888535534 us; error 42 us; skew 500 ppm
Aug 08 04:38:08 dist-test-slave-drl0 krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163888, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:38:08.539407 5628 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:38:08.540428 5628 webserver.cc:460] Webserver started at http://127.15.228.65:38787/ using document root <none> and password file <none>
I20260808 04:38:08.541071 5628 fs_manager.cc:359] Metadata directory not provided
I20260808 04:38:08.541177 5628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:38:08.541441 5628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:38:08.543798 5628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "0926e782ed534064b69801c09e7d58b0"
format_stamp: "Formatted at 2026-08-08 04:38:08 on dist-test-slave-drl0"
server_key: "9443282b05baf9d7342ee18065d52801"
server_key_iv: "b8d1aca71b308b768cc575a7c1d22c1f"
server_key_version: "encryptionkey@0"
I20260808 04:38:08.544420 5628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "0926e782ed534064b69801c09e7d58b0"
format_stamp: "Formatted at 2026-08-08 04:38:08 on dist-test-slave-drl0"
server_key: "9443282b05baf9d7342ee18065d52801"
server_key_iv: "b8d1aca71b308b768cc575a7c1d22c1f"
server_key_version: "encryptionkey@0"
I20260808 04:38:08.548460 5628 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:38:08.551685 5648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:08.553363 5628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:38:08.553493 5628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "0926e782ed534064b69801c09e7d58b0"
format_stamp: "Formatted at 2026-08-08 04:38:08 on dist-test-slave-drl0"
server_key: "9443282b05baf9d7342ee18065d52801"
server_key_iv: "b8d1aca71b308b768cc575a7c1d22c1f"
server_key_version: "encryptionkey@0"
I20260808 04:38:08.553580 5628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:38:08.577291 5628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:38:08.581909 5628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:38:08.582160 5628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:38:08.583421 5628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:38:08.584695 5628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:38:08.584764 5628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:08.584818 5628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:38:08.584836 5628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:08.597368 5628 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:34049
I20260808 04:38:08.597442 5807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:34049 every 8 connection(s)
I20260808 04:38:08.598472 5628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:38:08.607474 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 5628
I20260808 04:38:08.607661 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:38:08.607996 16273 external_mini_cluster.cc:1428] Setting key be6902012f90d3fd1e04cbaa4fff022b
Aug 08 04:38:08 dist-test-slave-drl0 krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163888, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:38:08.619577 5597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:08.600845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:33361 (local address 127.15.228.126:45107)
0808 04:38:08.606857 (+ 6012us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:08.606862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:08.606877 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:38:08.606970 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:08.606977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:08.607035 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:08.607134 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:38:08.609379 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:08.610057 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:08.611338 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:08.611570 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:08.611684 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:08.614468 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:38:08.614475 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:38:08.614482 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:38:08.614515 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:38:08.616678 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:08.617290 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:08.617295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:08.617299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:08.617402 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:08.617810 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:08.617815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:08.617817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:08.618084 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:38:08.618198 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:08.619051 (+ 853us) server_negotiation.cc:299] Negotiation successful
0808 04:38:08.619315 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5906}
WARNING: no policy specified for kudu/127.15.228.66@KRBTEST.COM; defaulting to no policy
I20260808 04:38:08.620430 5810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:08.601298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.65:33361)
0808 04:38:08.601971 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:08.602031 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 04:38:08.603682 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:38:08.607564 (+ 3882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:38:08.607576 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:38:08.608313 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:38:08.609200 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:08.609220 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:08.610194 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:08.610204 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:08.611206 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:08.611216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:08.611730 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:08.611734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:08.611910 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:08.612550 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:38:08.612568 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:38:08.614276 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:38:08.616833 (+ 2557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:38:08.616839 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:38:08.616854 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:38:08.617160 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:38:08.617537 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:38:08.617540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:38:08.617541 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:38:08.617687 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:38:08.618226 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:38:08.618231 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:38:08.618500 (+ 269us) client_negotiation.cc:769] Sending connection context
0808 04:38:08.618676 (+ 176us) client_negotiation.cc:240] Negotiation successful
0808 04:38:08.619054 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":172,"threads_started":1}
I20260808 04:38:08.622226 5808 heartbeater.cc:347] Connected to a master server at 127.15.228.126:45107
I20260808 04:38:08.622565 5808 heartbeater.cc:464] Registering TS with master...
I20260808 04:38:08.623152 5808 heartbeater.cc:511] Master 127.15.228.126:45107 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.66@KRBTEST.COM" created.
I20260808 04:38:08.625310 5541 ts_manager.cc:194] Registered new tserver with Master: 0926e782ed534064b69801c09e7d58b0 (127.15.228.65:34049)
I20260808 04:38:08.627966 5541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:33361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
I20260808 04:38:08.644285 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:45107
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:38:08.769279 5814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:38:08.769590 5814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:38:08.769666 5814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:38:08.773846 5814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:38:08.773936 5814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:38:08.774034 5814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:38:08.780457 5814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:38:08.781948 5814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:38:08.783408 5814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:38:08.794207 5824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:38:08.794193 5826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:38:08.794287 5828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:38:08.794710 5814 server_base.cc:1034] running on GCE node
I20260808 04:38:08.795228 5814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:38:08.796078 5814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:38:08.797315 5814 hybrid_clock.cc:648] HybridClock initialized: now 1786163888797284 us; error 58 us; skew 500 ppm
Aug 08 04:38:08 dist-test-slave-drl0 krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163888, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:38:08.800438 5814 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:38:08.801321 5814 webserver.cc:460] Webserver started at http://127.15.228.66:38889/ using document root <none> and password file <none>
I20260808 04:38:08.801846 5814 fs_manager.cc:359] Metadata directory not provided
I20260808 04:38:08.802003 5814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:38:08.802243 5814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:38:08.804528 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "578f0e1082ca4efebf13e84f1f256ba3"
format_stamp: "Formatted at 2026-08-08 04:38:08 on dist-test-slave-drl0"
server_key: "ed7b3a2d001b66f6bf10fc3d5cc9d9b8"
server_key_iv: "a260d79e5493808627211f8f69e59376"
server_key_version: "encryptionkey@0"
I20260808 04:38:08.805107 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "578f0e1082ca4efebf13e84f1f256ba3"
format_stamp: "Formatted at 2026-08-08 04:38:08 on dist-test-slave-drl0"
server_key: "ed7b3a2d001b66f6bf10fc3d5cc9d9b8"
server_key_iv: "a260d79e5493808627211f8f69e59376"
server_key_version: "encryptionkey@0"
I20260808 04:38:08.809209 5814 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:38:08.812271 5837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:08.813627 5814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:38:08.813791 5814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "578f0e1082ca4efebf13e84f1f256ba3"
format_stamp: "Formatted at 2026-08-08 04:38:08 on dist-test-slave-drl0"
server_key: "ed7b3a2d001b66f6bf10fc3d5cc9d9b8"
server_key_iv: "a260d79e5493808627211f8f69e59376"
server_key_version: "encryptionkey@0"
I20260808 04:38:08.813915 5814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:38:08.837999 5814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:38:08.841933 5814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:38:08.842181 5814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:38:08.842912 5814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:38:08.844058 5814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:38:08.844141 5814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:08.844220 5814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:38:08.844270 5814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:08.857105 5814 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:34977
I20260808 04:38:08.857131 5997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:34977 every 8 connection(s)
I20260808 04:38:08.858067 5814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:38:08.863308 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 5814
I20260808 04:38:08.863493 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:38:08.863916 16273 external_mini_cluster.cc:1428] Setting key c75110072a314cdc953ad61776e3f392
Aug 08 04:38:08 dist-test-slave-drl0 krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163888, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:38:08.878793 5597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:08.860225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:45187 (local address 127.15.228.126:45107)
0808 04:38:08.860560 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:08.860568 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:08.861840 (+ 1272us) server_negotiation.cc:407] Connection header received
0808 04:38:08.863743 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:08.863749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:08.863834 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:08.863970 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:38:08.866218 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:08.867186 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:08.868540 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:08.868953 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:08.869084 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:08.873072 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:38:08.873077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:38:08.873079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:38:08.873111 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:38:08.875208 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:08.876063 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:08.876066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:08.876068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:08.876151 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:08.876662 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:08.876667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:08.876668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:08.876962 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:38:08.877092 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:08.878382 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0808 04:38:08.878645 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
WARNING: no policy specified for kudu/127.15.228.67@KRBTEST.COM; defaulting to no policy
I20260808 04:38:08.880057 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:08.860860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.66:45187)
0808 04:38:08.861586 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:08.861656 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0808 04:38:08.863329 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:38:08.864259 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:38:08.864266 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:38:08.865042 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:38:08.865893 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:08.865917 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:08.867394 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:08.867398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:08.868369 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:08.868381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:08.869230 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:08.869238 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:08.869690 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:08.870629 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:38:08.870660 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:38:08.872822 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:38:08.875394 (+ 2572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:38:08.875405 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:38:08.875431 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:38:08.875927 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:38:08.876262 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:38:08.876265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:38:08.876267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:38:08.876482 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:38:08.877252 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:38:08.877267 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:38:08.877867 (+ 600us) client_negotiation.cc:769] Sending connection context
0808 04:38:08.878303 (+ 436us) client_negotiation.cc:240] Negotiation successful
0808 04:38:08.878918 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":124,"threads_started":1}
I20260808 04:38:08.881508 6000 heartbeater.cc:347] Connected to a master server at 127.15.228.126:45107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.67@KRBTEST.COM" created.
I20260808 04:38:08.881800 6000 heartbeater.cc:464] Registering TS with master...
I20260808 04:38:08.882510 6000 heartbeater.cc:511] Master 127.15.228.126:45107 requested a full tablet report, sending...
I20260808 04:38:08.883857 5541 ts_manager.cc:194] Registered new tserver with Master: 578f0e1082ca4efebf13e84f1f256ba3 (127.15.228.66:34977)
I20260808 04:38:08.884665 5541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:45187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
I20260808 04:38:08.902994 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:45107
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:38:09.037086 6005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:38:09.037401 6005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:38:09.037487 6005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:38:09.041618 6005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:38:09.041718 6005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:38:09.041811 6005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:38:09.047653 6005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:38:09.048808 6005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:38:09.049904 6005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:38:09.060763 6014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:38:09.060786 6012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:38:09.060708 6011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:38:09.060971 6005 server_base.cc:1034] running on GCE node
I20260808 04:38:09.061470 6005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:38:09.062125 6005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:38:09.063347 6005 hybrid_clock.cc:648] HybridClock initialized: now 1786163889063319 us; error 39 us; skew 500 ppm
Aug 08 04:38:09 dist-test-slave-drl0 krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163889, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:38:09.066880 6005 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:38:09.067842 6005 webserver.cc:460] Webserver started at http://127.15.228.67:33263/ using document root <none> and password file <none>
I20260808 04:38:09.068388 6005 fs_manager.cc:359] Metadata directory not provided
I20260808 04:38:09.068449 6005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:38:09.068612 6005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:38:09.070629 6005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "d389e52d8a484c4891390fbeb7fca81c"
format_stamp: "Formatted at 2026-08-08 04:38:09 on dist-test-slave-drl0"
server_key: "162c24eac55a998ecf211d292d162abf"
server_key_iv: "db6928419f1a614db8917e96b0dea095"
server_key_version: "encryptionkey@0"
I20260808 04:38:09.071220 6005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "d389e52d8a484c4891390fbeb7fca81c"
format_stamp: "Formatted at 2026-08-08 04:38:09 on dist-test-slave-drl0"
server_key: "162c24eac55a998ecf211d292d162abf"
server_key_iv: "db6928419f1a614db8917e96b0dea095"
server_key_version: "encryptionkey@0"
I20260808 04:38:09.075712 6005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:38:09.079258 6023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:09.081008 6005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:38:09.081154 6005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "d389e52d8a484c4891390fbeb7fca81c"
format_stamp: "Formatted at 2026-08-08 04:38:09 on dist-test-slave-drl0"
server_key: "162c24eac55a998ecf211d292d162abf"
server_key_iv: "db6928419f1a614db8917e96b0dea095"
server_key_version: "encryptionkey@0"
I20260808 04:38:09.081290 6005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:38:09.113564 6005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:38:09.117339 6005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:38:09.117591 6005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:38:09.118270 6005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:38:09.119295 6005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:38:09.119360 6005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:09.119438 6005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:38:09.119481 6005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:09.129972 6005 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:40449
I20260808 04:38:09.129997 6185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:40449 every 8 connection(s)
I20260808 04:38:09.131052 6005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:38:09.133409 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 6005
I20260808 04:38:09.133569 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:38:09.133869 16273 external_mini_cluster.cc:1428] Setting key 3c060ec0ef70b3a4e50b3703073c0095
Aug 08 04:38:09 dist-test-slave-drl0 krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163889, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:38:09.148247 5597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:09.133153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:49773 (local address 127.15.228.126:45107)
0808 04:38:09.133332 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:09.133337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:09.134479 (+ 1142us) server_negotiation.cc:407] Connection header received
0808 04:38:09.136467 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:09.136470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:09.136532 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:09.136621 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:38:09.137897 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.138446 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.139811 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.140038 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.140129 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:09.143462 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:38:09.143466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:38:09.143468 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:38:09.143491 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:38:09.145213 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:09.145814 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:09.145817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:09.145818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:09.145907 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:09.146397 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:09.146402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:09.146403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:09.146704 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:38:09.146881 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:09.147927 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0808 04:38:09.148068 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 04:38:09.149202 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:09.133727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.67:49773)
0808 04:38:09.134318 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:09.134361 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 04:38:09.136189 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:38:09.136809 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:38:09.136816 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:38:09.137241 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:38:09.137739 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:09.137752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.138586 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.138590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:09.139676 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:09.139687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.140205 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.140208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:09.140365 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:09.140990 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:38:09.141007 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:38:09.143203 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:38:09.145360 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:38:09.145365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:38:09.145387 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:38:09.145678 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:38:09.146024 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:38:09.146026 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:38:09.146028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:38:09.146189 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:38:09.146891 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:38:09.146903 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:38:09.147480 (+ 577us) client_negotiation.cc:769] Sending connection context
0808 04:38:09.147863 (+ 383us) client_negotiation.cc:240] Negotiation successful
0808 04:38:09.148232 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":172,"threads_started":1}
I20260808 04:38:09.150671 6186 heartbeater.cc:347] Connected to a master server at 127.15.228.126:45107
I20260808 04:38:09.151016 6186 heartbeater.cc:464] Registering TS with master...
I20260808 04:38:09.151551 6186 heartbeater.cc:511] Master 127.15.228.126:45107 requested a full tablet report, sending...
I20260808 04:38:09.152693 5541 ts_manager.cc:194] Registered new tserver with Master: d389e52d8a484c4891390fbeb7fca81c (127.15.228.67:40449)
I20260808 04:38:09.153293 5541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:49773
I20260808 04:38:09.161759 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:38:09.173748 5597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:09.164437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.15.228.126:45107)
0808 04:38:09.164597 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:09.164600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:09.164837 (+ 237us) server_negotiation.cc:407] Connection header received
0808 04:38:09.165301 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:09.165304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:09.165351 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:09.165438 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:38:09.166278 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.166817 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.167833 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.168234 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.168357 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:09.170076 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:38:09.170079 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:38:09.170081 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:38:09.170103 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:38:09.171823 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:09.172440 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:09.172443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:09.172444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:09.172522 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:09.172889 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:09.172892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:09.172893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:09.173121 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:38:09.173231 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:09.173510 (+ 279us) server_negotiation.cc:299] Negotiation successful
0808 04:38:09.173621 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 04:38:09.177678 5541 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:38:09.181012 5541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:38:09.200837 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:09.193758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:34977 (local address 127.0.0.1:36696)
0808 04:38:09.194463 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:09.194479 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:38:09.194570 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:38:09.195740 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:38:09.195745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:38:09.195768 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:38:09.195843 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:09.195849 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.196885 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.196890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:09.198473 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:09.198480 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.199678 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.199684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:09.199968 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:09.200142 (+ 174us) client_negotiation.cc:769] Sending connection context
0808 04:38:09.200345 (+ 203us) client_negotiation.cc:240] Negotiation successful
0808 04:38:09.200594 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":235,"thread_start_us":137,"threads_started":1}
I20260808 04:38:09.200837 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:09.194393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:34049 (local address 127.0.0.1:59972)
0808 04:38:09.194911 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:09.194952 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:38:09.195044 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:38:09.195866 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:38:09.195869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:38:09.195887 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:38:09.195985 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:09.195991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.197407 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.197411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:09.198392 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:09.198402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.199678 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.199684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:09.199953 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:09.200142 (+ 189us) client_negotiation.cc:769] Sending connection context
0808 04:38:09.200345 (+ 203us) client_negotiation.cc:240] Negotiation successful
0808 04:38:09.200585 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":62,"thread_start_us":91,"threads_started":1}
I20260808 04:38:09.201364 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:09.193659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.15.228.67:40449)
0808 04:38:09.194815 (+ 1156us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:09.194828 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:09.194851 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:38:09.194977 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:09.194984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:09.195203 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:09.195383 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:38:09.195698 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.196964 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.199741 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.200262 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.200403 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:09.200535 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:09.201098 (+ 563us) server_negotiation.cc:299] Negotiation successful
0808 04:38:09.201206 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1050,"thread_start_us":142,"threads_started":1}
I20260808 04:38:09.201483 6205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:09.194823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59972 (local address 127.15.228.65:34049)
0808 04:38:09.195340 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:09.195347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:09.195367 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:38:09.195432 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:09.195436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:09.195580 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:09.195720 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:38:09.196115 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.197244 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.198522 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.199399 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.199523 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:09.199677 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:09.200405 (+ 728us) server_negotiation.cc:299] Negotiation successful
0808 04:38:09.201315 (+ 910us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":150,"threads_started":1}
I20260808 04:38:09.201231 6198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:09.193475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:40449 (local address 127.0.0.1:35510)
0808 04:38:09.194311 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:09.194367 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 04:38:09.194505 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:38:09.195373 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:38:09.195388 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:38:09.195412 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:38:09.195577 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:09.195584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.197106 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.197109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:09.198467 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:09.198480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.200609 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.200614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:09.200723 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:09.200753 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 04:38:09.200825 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 04:38:09.200915 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":57,"thread_start_us":95,"threads_started":1}
I20260808 04:38:09.202008 6203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:09.193949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36696 (local address 127.15.228.66:34977)
0808 04:38:09.195284 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:09.195296 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:09.195318 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:38:09.195399 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:09.195403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:09.195573 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:09.195756 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:38:09.195946 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.196733 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.198575 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.199401 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.199523 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:09.199677 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:09.201724 (+ 2047us) server_negotiation.cc:299] Negotiation successful
0808 04:38:09.201853 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1234,"thread_start_us":163,"threads_started":1}
I20260808 04:38:09.205417 5711 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6c909858a4479790f322664f43705b (DEFAULT_TABLE table=test-table [id=9057d84288234f97bb0c574860317a11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:38:09.205782 6087 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6c909858a4479790f322664f43705b (DEFAULT_TABLE table=test-table [id=9057d84288234f97bb0c574860317a11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:38:09.205785 5906 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6c909858a4479790f322664f43705b (DEFAULT_TABLE table=test-table [id=9057d84288234f97bb0c574860317a11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:38:09.206487 5711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6c909858a4479790f322664f43705b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:38:09.207300 6087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6c909858a4479790f322664f43705b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:38:09.207300 5906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6c909858a4479790f322664f43705b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:38:09.213109 6206 tablet_bootstrap.cc:492] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0: Bootstrap starting.
I20260808 04:38:09.215607 6207 tablet_bootstrap.cc:492] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3: Bootstrap starting.
I20260808 04:38:09.215607 6209 tablet_bootstrap.cc:492] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c: Bootstrap starting.
I20260808 04:38:09.216405 6206 tablet_bootstrap.cc:654] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0: Neither blocks nor log segments found. Creating new log.
I20260808 04:38:09.217331 6206 log.cc:824] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0: Log is configured to *not* fsync() on all Append() calls
I20260808 04:38:09.219558 6206 tablet_bootstrap.cc:492] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0: No bootstrap required, opened a new log
I20260808 04:38:09.219805 6206 ts_tablet_manager.cc:1397] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260808 04:38:09.219993 6207 tablet_bootstrap.cc:654] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3: Neither blocks nor log segments found. Creating new log.
I20260808 04:38:09.219993 6209 tablet_bootstrap.cc:654] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c: Neither blocks nor log segments found. Creating new log.
I20260808 04:38:09.220918 6209 log.cc:824] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c: Log is configured to *not* fsync() on all Append() calls
I20260808 04:38:09.220914 6207 log.cc:824] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3: Log is configured to *not* fsync() on all Append() calls
I20260808 04:38:09.223634 6207 tablet_bootstrap.cc:492] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3: No bootstrap required, opened a new log
I20260808 04:38:09.223635 6209 tablet_bootstrap.cc:492] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c: No bootstrap required, opened a new log
I20260808 04:38:09.223858 6209 ts_tablet_manager.cc:1397] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 04:38:09.223858 6207 ts_tablet_manager.cc:1397] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260808 04:38:09.224663 6206 raft_consensus.cc:348] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:09.224927 6206 raft_consensus.cc:374] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:38:09.225003 6206 raft_consensus.cc:729] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0926e782ed534064b69801c09e7d58b0, State: Initialized, Role: FOLLOWER
I20260808 04:38:09.225512 6206 consensus_queue.cc:260] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:09.226246 5808 heartbeater.cc:503] Master 127.15.228.126:45107 was elected leader, sending a full tablet report...
I20260808 04:38:09.226572 6206 ts_tablet_manager.cc:1428] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260808 04:38:09.227694 6209 raft_consensus.cc:348] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:09.227980 6209 raft_consensus.cc:374] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:38:09.228029 6209 raft_consensus.cc:729] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d389e52d8a484c4891390fbeb7fca81c, State: Initialized, Role: FOLLOWER
I20260808 04:38:09.228497 6209 consensus_queue.cc:260] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:09.228917 6207 raft_consensus.cc:348] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:09.229369 6186 heartbeater.cc:503] Master 127.15.228.126:45107 was elected leader, sending a full tablet report...
I20260808 04:38:09.229483 6209 ts_tablet_manager.cc:1428] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:38:09.230230 6207 raft_consensus.cc:374] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:38:09.230324 6207 raft_consensus.cc:729] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 578f0e1082ca4efebf13e84f1f256ba3, State: Initialized, Role: FOLLOWER
I20260808 04:38:09.230793 6207 consensus_queue.cc:260] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:09.231817 6207 ts_tablet_manager.cc:1428] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260808 04:38:09.232712 6000 heartbeater.cc:503] Master 127.15.228.126:45107 was elected leader, sending a full tablet report...
W20260808 04:38:09.350203 5809 tablet.cc:2369] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:38:09.359547 6001 tablet.cc:2369] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:38:09.382258 6188 tablet.cc:2369] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:38:09.445847 6213 raft_consensus.cc:482] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:38:09.446241 6213 raft_consensus.cc:504] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:09.447805 6213 leader_election.cc:288] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d389e52d8a484c4891390fbeb7fca81c (127.15.228.67:40449), 578f0e1082ca4efebf13e84f1f256ba3 (127.15.228.66:34977)
I20260808 04:38:09.452404 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:09.448346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:59321 (local address 127.15.228.67:40449)
0808 04:38:09.448547 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:09.448556 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:09.448885 (+ 329us) server_negotiation.cc:407] Connection header received
0808 04:38:09.449074 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:09.449078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:09.449137 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:09.449224 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:38:09.449540 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.450065 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.451382 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.451794 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.451938 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:09.452000 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:09.452197 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 04:38:09.452270 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 04:38:09.452419 6216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:09.448270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:40449 (local address 127.15.228.65:59321)
0808 04:38:09.448744 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:09.448757 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:38:09.448939 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:38:09.449237 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:38:09.449240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:38:09.449257 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:38:09.449422 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:09.449428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.450185 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.450190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:09.451158 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:09.451167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.451937 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.451941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:09.452045 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:09.452068 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:38:09.452166 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 04:38:09.452244 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":139,"threads_started":1}
I20260808 04:38:09.452682 6203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:09.448375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:58929 (local address 127.15.228.66:34977)
0808 04:38:09.448532 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:09.448538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:09.449092 (+ 554us) server_negotiation.cc:407] Connection header received
0808 04:38:09.449422 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:09.449427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:09.449493 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:09.449594 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:38:09.450008 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.450620 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.451733 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.452127 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.452271 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:09.452326 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:09.452499 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 04:38:09.452570 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:38:09.452715 6217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:09.448270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:34977 (local address 127.15.228.65:58929)
0808 04:38:09.448892 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:09.448906 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:38:09.449172 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:38:09.449609 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:38:09.449612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:38:09.449632 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:38:09.449848 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:09.449860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.450783 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.450788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:09.451595 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:09.451604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.452257 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.452261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:09.452361 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:09.452388 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 04:38:09.452479 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 04:38:09.452548 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":45,"thread_start_us":204,"threads_started":1}
I20260808 04:38:09.453284 6118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6c909858a4479790f322664f43705b" candidate_uuid: "0926e782ed534064b69801c09e7d58b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d389e52d8a484c4891390fbeb7fca81c" is_pre_election: true
I20260808 04:38:09.453397 5932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6c909858a4479790f322664f43705b" candidate_uuid: "0926e782ed534064b69801c09e7d58b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "578f0e1082ca4efebf13e84f1f256ba3" is_pre_election: true
I20260808 04:38:09.453676 6118 raft_consensus.cc:2454] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0926e782ed534064b69801c09e7d58b0 in term 0.
I20260808 04:38:09.453994 5932 raft_consensus.cc:2454] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0926e782ed534064b69801c09e7d58b0 in term 0.
I20260808 04:38:09.454262 5653 leader_election.cc:302] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0926e782ed534064b69801c09e7d58b0, d389e52d8a484c4891390fbeb7fca81c; no voters:
I20260808 04:38:09.454619 6213 raft_consensus.cc:2781] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:38:09.454721 6213 raft_consensus.cc:482] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:38:09.454777 6213 raft_consensus.cc:3037] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:38:09.455966 6213 raft_consensus.cc:504] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:09.456374 6213 leader_election.cc:288] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 1 election: Requested vote from peers d389e52d8a484c4891390fbeb7fca81c (127.15.228.67:40449), 578f0e1082ca4efebf13e84f1f256ba3 (127.15.228.66:34977)
I20260808 04:38:09.456749 6118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6c909858a4479790f322664f43705b" candidate_uuid: "0926e782ed534064b69801c09e7d58b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d389e52d8a484c4891390fbeb7fca81c"
I20260808 04:38:09.456848 5932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6c909858a4479790f322664f43705b" candidate_uuid: "0926e782ed534064b69801c09e7d58b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "578f0e1082ca4efebf13e84f1f256ba3"
I20260808 04:38:09.456904 6118 raft_consensus.cc:3037] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:38:09.456996 5932 raft_consensus.cc:3037] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:38:09.458024 6118 raft_consensus.cc:2454] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0926e782ed534064b69801c09e7d58b0 in term 1.
I20260808 04:38:09.458215 5932 raft_consensus.cc:2454] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0926e782ed534064b69801c09e7d58b0 in term 1.
I20260808 04:38:09.458361 5653 leader_election.cc:302] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0926e782ed534064b69801c09e7d58b0, d389e52d8a484c4891390fbeb7fca81c; no voters:
I20260808 04:38:09.458590 6213 raft_consensus.cc:2781] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:38:09.458909 6213 raft_consensus.cc:686] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [term 1 LEADER]: Becoming Leader. State: Replica: 0926e782ed534064b69801c09e7d58b0, State: Running, Role: LEADER
I20260808 04:38:09.459394 6213 consensus_queue.cc:237] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:09.463299 5541 catalog_manager.cc:5495] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0926e782ed534064b69801c09e7d58b0 (127.15.228.65). New cstate: current_term: 1 leader_uuid: "0926e782ed534064b69801c09e7d58b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } health_report { overall_health: HEALTHY } } }
I20260808 04:38:09.496474 6205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:09.492749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.15.228.65:34049)
0808 04:38:09.492908 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:09.492913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:09.492931 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:38:09.493036 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:09.493041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:09.493106 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:09.493225 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:38:09.493489 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.494014 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.494913 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:09.495162 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:09.495283 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:09.495607 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:38:09.496036 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:38:09.496178 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:09.496250 (+ 72us) server_negotiation.cc:299] Negotiation successful
0808 04:38:09.496307 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 04:38:09.503203 5932 raft_consensus.cc:1261] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3 [term 1 FOLLOWER]: Refusing update from remote peer 0926e782ed534064b69801c09e7d58b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:38:09.503203 6118 raft_consensus.cc:1261] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c [term 1 FOLLOWER]: Refusing update from remote peer 0926e782ed534064b69801c09e7d58b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:38:09.504002 6213 consensus_queue.cc:1035] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:38:09.504204 6218 consensus_queue.cc:1035] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:38:09.515587 6226 mvcc.cc:205] Tried to move back new op lower bound from 7316127291396648960 to 7316127291228868608. Current Snapshot: MvccSnapshot[applied={T|T < 7316127291396648960}]
I20260808 04:38:09.516690 6227 mvcc.cc:205] Tried to move back new op lower bound from 7316127291396648960 to 7316127291228868608. Current Snapshot: MvccSnapshot[applied={T|T < 7316127291396648960}]
I20260808 04:38:09.523945 5540 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:38:09.524236 5540 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:38:09.526803 5540 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99: Sending DeleteTablet for 3 replicas of tablet 2d6c909858a4479790f322664f43705b
I20260808 04:38:09.527650 6087 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d6c909858a4479790f322664f43705b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:38:09 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:35510
I20260808 04:38:09.527768 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 5628
I20260808 04:38:09.527765 5711 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d6c909858a4479790f322664f43705b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:38:09 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:59972
I20260808 04:38:09.531647 5906 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d6c909858a4479790f322664f43705b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:38:09 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:36696
I20260808 04:38:09.532996 6238 tablet_replica.cc:330] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3: stopping tablet replica
I20260808 04:38:09.534796 6237 tablet_replica.cc:330] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c: stopping tablet replica
I20260808 04:38:09.534818 6238 raft_consensus.cc:2229] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:38:09.535243 6237 raft_consensus.cc:2229] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:38:09.535377 6238 raft_consensus.cc:2258] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:38:09.535607 6237 raft_consensus.cc:2258] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:38:09.538443 6237 ts_tablet_manager.cc:1905] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:38:09.538460 6238 ts_tablet_manager.cc:1905] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:38:09.541062 6237 ts_tablet_manager.cc:1918] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:38:09.541203 6237 log.cc:1196] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/2d6c909858a4479790f322664f43705b
I20260808 04:38:09.541611 6237 ts_tablet_manager.cc:1939] T 2d6c909858a4479790f322664f43705b P d389e52d8a484c4891390fbeb7fca81c: Deleting consensus metadata
I20260808 04:38:09.542148 6238 ts_tablet_manager.cc:1918] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:38:09.542260 6238 log.cc:1196] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/2d6c909858a4479790f322664f43705b
I20260808 04:38:09.542666 6238 ts_tablet_manager.cc:1939] T 2d6c909858a4479790f322664f43705b P 578f0e1082ca4efebf13e84f1f256ba3: Deleting consensus metadata
I20260808 04:38:09.543185 5521 catalog_manager.cc:4841] TS d389e52d8a484c4891390fbeb7fca81c (127.15.228.67:40449): tablet 2d6c909858a4479790f322664f43705b (table test-table [id=9057d84288234f97bb0c574860317a11]) successfully deleted
I20260808 04:38:09.543900 5520 catalog_manager.cc:4841] TS 578f0e1082ca4efebf13e84f1f256ba3 (127.15.228.66:34977): tablet 2d6c909858a4479790f322664f43705b (table test-table [id=9057d84288234f97bb0c574860317a11]) successfully deleted
W20260808 04:38:09.545360 5520 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.228.65:34049 (error 108)
I20260808 04:38:09.546284 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 5814
I20260808 04:38:09.546546 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:09.546200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:34049 (local address 127.0.0.1:59990)
0808 04:38:09.546343 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:09.546445 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.65:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 04:38:09.552802 5520 catalog_manager.cc:4584] TS 0926e782ed534064b69801c09e7d58b0 (127.15.228.65:34049): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d6c909858a4479790f322664f43705b: Network error: Client connection negotiation failed: client connection to 127.15.228.65:34049: connect: Connection refused (error 111)
I20260808 04:38:09.560253 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 6005
I20260808 04:38:09.579929 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 5491
I20260808 04:38:29.594851 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:45107
--webserver_interface=127.15.228.126
--webserver_port=45783
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:45107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:38:29.725371 6240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:38:29.725718 6240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:38:29.725770 6240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:38:29.730494 6240 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 04:38:29.730564 6240 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 04:38:29.730582 6240 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 04:38:29.730597 6240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:38:29.730618 6240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:38:29.730638 6240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:38:29.730655 6240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:38:29.730672 6240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:38:29.736286 6240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:45107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:45107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=45783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:38:29.737442 6240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:38:29.738525 6240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:38:29.756965 6240 server_base.cc:1034] running on GCE node
W20260808 04:38:29.757117 6251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:38:29.757156 6249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:38:29.757218 6248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:38:29.758236 6240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:38:29.759609 6240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:38:29.760905 6240 hybrid_clock.cc:648] HybridClock initialized: now 1786163909760850 us; error 47 us; skew 500 ppm
Aug 08 04:38:29 dist-test-slave-drl0 krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163909, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:38:29.770170 6240 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:38:29.772043 6240 webserver.cc:460] Webserver started at http://127.15.228.126:45783/ using document root <none> and password file <none>
I20260808 04:38:29.772907 6240 fs_manager.cc:359] Metadata directory not provided
I20260808 04:38:29.773051 6240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:38:29.778774 6240 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260808 04:38:29.781895 6268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:29.783562 6240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:38:29.783682 6240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "5025a708d2244712a14fe71c6895fe99"
format_stamp: "Formatted at 2026-08-08 04:38:08 on dist-test-slave-drl0"
server_key: "fd46ca6520eaf2271a69f50cf2ea5979"
server_key_iv: "23ae778879d5bae8503a56739fedf955"
server_key_version: "encryptionkey@0"
I20260808 04:38:29.784382 6240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:38:29.802768 6240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:38:29.806548 6240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:38:29.806734 6240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:38:29.816936 6240 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:45107
I20260808 04:38:29.817313 6339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:45107 every 8 connection(s)
I20260808 04:38:29.818440 6240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:38:29.824527 6341 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:38:29.825438 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 6240
I20260808 04:38:29.826149 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:34049
--local_ip_for_outbound_sockets=127.15.228.65
--tserver_master_addrs=127.15.228.126:45107
--webserver_port=38787
--webserver_interface=127.15.228.65
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:38:29.827281 6341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99: Bootstrap starting.
I20260808 04:38:29.856658 6341 log.cc:824] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99: Log is configured to *not* fsync() on all Append() calls
I20260808 04:38:29.870887 6341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:38:29.871439 6341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99: Bootstrap complete.
I20260808 04:38:29.878226 6341 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5025a708d2244712a14fe71c6895fe99" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45107 } }
I20260808 04:38:29.878768 6341 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5025a708d2244712a14fe71c6895fe99, State: Initialized, Role: FOLLOWER
I20260808 04:38:29.879390 6341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5025a708d2244712a14fe71c6895fe99" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45107 } }
I20260808 04:38:29.879554 6341 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:38:29.879624 6341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:38:29.879727 6341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 1 FOLLOWER]: Advancing to term 2
I20260808 04:38:29.882452 6341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5025a708d2244712a14fe71c6895fe99" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45107 } }
I20260808 04:38:29.882851 6341 leader_election.cc:302] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5025a708d2244712a14fe71c6895fe99; no voters:
I20260808 04:38:29.883313 6341 leader_election.cc:288] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 04:38:29.883701 6348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 04:38:29.884253 6348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 2 LEADER]: Becoming Leader. State: Replica: 5025a708d2244712a14fe71c6895fe99, State: Running, Role: LEADER
I20260808 04:38:29.884652 6348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5025a708d2244712a14fe71c6895fe99" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45107 } }
I20260808 04:38:29.884663 6341 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:38:29.886324 6349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5025a708d2244712a14fe71c6895fe99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5025a708d2244712a14fe71c6895fe99" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45107 } } }
I20260808 04:38:29.886373 6350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5025a708d2244712a14fe71c6895fe99. Latest consensus state: current_term: 2 leader_uuid: "5025a708d2244712a14fe71c6895fe99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5025a708d2244712a14fe71c6895fe99" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45107 } } }
I20260808 04:38:29.886467 6349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [sys.catalog]: This master's current role is: LEADER
I20260808 04:38:29.886472 6350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [sys.catalog]: This master's current role is: LEADER
I20260808 04:38:29.887066 6362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:38:29.891866 6362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:38:29.892411 6362 catalog_manager.cc:1259] Loaded cluster ID: 53b08da430a7494c97ae706acd573c3b
I20260808 04:38:29.892488 6362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:38:29.894356 6362 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:38:29.895080 6362 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99: Loaded TSK: 0
I20260808 04:38:29.918568 6362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99: Generated new TSK 1
I20260808 04:38:29.918922 6362 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:38:29.992919 6344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:38:29.993333 6344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:38:29.993396 6344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:38:29.998659 6344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:38:29.998819 6344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:38:29.999002 6344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:38:30.004658 6344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:34049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=38787
--tserver_master_addrs=127.15.228.126:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:38:30.005738 6344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:38:30.006805 6344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:38:30.019536 6377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:38:30.019848 6380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:38:30.019541 6378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:38:30.022554 6344 server_base.cc:1034] running on GCE node
I20260808 04:38:30.023169 6344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:38:30.024101 6344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:38:30.025323 6344 hybrid_clock.cc:648] HybridClock initialized: now 1786163910025272 us; error 64 us; skew 500 ppm
Aug 08 04:38:30 dist-test-slave-drl0 krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163910, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:38:30.029815 6344 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:38:30.031448 6344 webserver.cc:460] Webserver started at http://127.15.228.65:38787/ using document root <none> and password file <none>
I20260808 04:38:30.032449 6344 fs_manager.cc:359] Metadata directory not provided
I20260808 04:38:30.032599 6344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:38:30.039799 6344 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:38:30.043352 6388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:30.045106 6344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:38:30.045240 6344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "0926e782ed534064b69801c09e7d58b0"
format_stamp: "Formatted at 2026-08-08 04:38:08 on dist-test-slave-drl0"
server_key: "9443282b05baf9d7342ee18065d52801"
server_key_iv: "b8d1aca71b308b768cc575a7c1d22c1f"
server_key_version: "encryptionkey@0"
I20260808 04:38:30.046195 6344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:38:30.065408 6344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:38:30.069541 6344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:38:30.069747 6344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:38:30.070454 6344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:38:30.072018 6400 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 04:38:30.076300 6344 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 04:38:30.076390 6344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260808 04:38:30.076450 6344 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 04:38:30.079030 6344 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 04:38:30.079098 6344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260808 04:38:30.080147 6400 tablet_bootstrap.cc:492] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0: Bootstrap starting.
I20260808 04:38:30.096371 6344 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:34049
I20260808 04:38:30.096580 6545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:34049 every 8 connection(s)
I20260808 04:38:30.097486 6344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:38:30.101827 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 6344
I20260808 04:38:30.102332 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:34977
--local_ip_for_outbound_sockets=127.15.228.66
--tserver_master_addrs=127.15.228.126:45107
--webserver_port=38889
--webserver_interface=127.15.228.66
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:38:30.105878 6400 log.cc:824] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:38:30 dist-test-slave-drl0 krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163910, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:38:30.114871 6400 tablet_bootstrap.cc:492] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:38:30.115337 6400 tablet_bootstrap.cc:492] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0: Bootstrap complete.
I20260808 04:38:30.116937 6400 ts_tablet_manager.cc:1397] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0: Time spent bootstrapping tablet: real 0.037s user 0.015s sys 0.019s
I20260808 04:38:30.119557 6549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.099900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:37921 (local address 127.15.228.126:45107)
0808 04:38:30.100549 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:30.100572 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:30.100630 (+ 58us) server_negotiation.cc:407] Connection header received
0808 04:38:30.101825 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:30.101844 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:30.102288 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:30.102674 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:38:30.104889 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.105883 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.107715 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.108403 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.108576 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:30.112430 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:38:30.112440 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:38:30.112463 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:38:30.112508 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:38:30.115389 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:30.116141 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:30.116151 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:30.116160 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:30.116354 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:30.116809 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:30.116815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:30.116817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:30.117376 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:38:30.117552 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:30.118206 (+ 654us) server_negotiation.cc:299] Negotiation successful
0808 04:38:30.118625 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":159,"threads_started":1}
I20260808 04:38:30.119601 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.099866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.65:37921)
0808 04:38:30.100387 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:30.100445 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 04:38:30.101319 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:38:30.103175 (+ 1856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:38:30.103187 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:38:30.103947 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:38:30.104666 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:30.104693 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.106074 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.106080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:30.107554 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:30.107570 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.108605 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.108612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:30.108864 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:30.109789 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:38:30.109829 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:38:30.112178 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:38:30.115541 (+ 3363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:38:30.115547 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:38:30.115560 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:38:30.115981 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:38:30.116478 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:38:30.116484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:38:30.116489 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:38:30.116677 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:38:30.117530 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:38:30.117536 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:38:30.117836 (+ 300us) client_negotiation.cc:769] Sending connection context
0808 04:38:30.118028 (+ 192us) client_negotiation.cc:240] Negotiation successful
0808 04:38:30.118377 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":155,"threads_started":1}
I20260808 04:38:30.121033 6400 raft_consensus.cc:348] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:30.121842 6400 raft_consensus.cc:729] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0926e782ed534064b69801c09e7d58b0, State: Initialized, Role: FOLLOWER
I20260808 04:38:30.122746 6400 consensus_queue.cc:260] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:30.123961 6400 ts_tablet_manager.cc:1428] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260808 04:38:30.124115 6546 heartbeater.cc:347] Connected to a master server at 127.15.228.126:45107
I20260808 04:38:30.124382 6546 heartbeater.cc:464] Registering TS with master...
I20260808 04:38:30.124993 6546 heartbeater.cc:511] Master 127.15.228.126:45107 requested a full tablet report, sending...
I20260808 04:38:30.128160 6295 ts_manager.cc:194] Registered new tserver with Master: 0926e782ed534064b69801c09e7d58b0 (127.15.228.65:34049)
I20260808 04:38:30.132434 6295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:37921
I20260808 04:38:30.135656 6546 heartbeater.cc:503] Master 127.15.228.126:45107 was elected leader, sending a full tablet report...
Aug 08 04:38:30 dist-test-slave-drl0 krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163909, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for kudu/127.15.228.65@KRBTEST.COM
I20260808 04:38:30.144829 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.132485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41266 (local address 127.15.228.65:34049)
0808 04:38:30.132960 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:30.132976 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:30.133001 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:38:30.133073 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:30.133078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:30.133291 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:30.133467 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:38:30.134325 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.135207 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.136419 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.136856 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.137051 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:30.139658 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:38:30.139667 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:38:30.139677 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:38:30.139700 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:38:30.141825 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:30.142631 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:30.142638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:30.142640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:30.142746 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:30.143168 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:30.143172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:30.143174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:30.143540 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:38:30.143853 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:30.144345 (+ 492us) server_negotiation.cc:299] Negotiation successful
0808 04:38:30.144585 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":130,"threads_started":1}
I20260808 04:38:30.144829 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.131768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:34049 (local address 127.0.0.1:41266)
0808 04:38:30.132220 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:30.132270 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 04:38:30.132427 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:38:30.133653 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:38:30.133659 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:38:30.134062 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:38:30.134173 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:30.134180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.135355 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.135362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:30.136267 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:30.136278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.137065 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.137070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:30.137225 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:30.137995 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:38:30.138008 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:38:30.139507 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:38:30.142073 (+ 2566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:38:30.142096 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:38:30.142104 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:38:30.142448 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:38:30.142871 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:38:30.142875 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:38:30.142877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:38:30.143051 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:38:30.143803 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:38:30.143818 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:38:30.144010 (+ 192us) client_negotiation.cc:769] Sending connection context
0808 04:38:30.144223 (+ 213us) client_negotiation.cc:240] Negotiation successful
0808 04:38:30.144585 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":117,"threads_started":1}
I20260808 04:38:30.146345 6453 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d6c909858a4479790f322664f43705b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:38:09 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:41266
I20260808 04:38:30.147607 6558 tablet_replica.cc:330] stopping tablet replica
I20260808 04:38:30.148063 6558 raft_consensus.cc:2229] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:38:30.148348 6558 raft_consensus.cc:2258] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:38:30.149506 6558 ts_tablet_manager.cc:1905] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:38:30.153687 6558 ts_tablet_manager.cc:1918] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:38:30.153905 6558 log.cc:1196] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/wal/wals/2d6c909858a4479790f322664f43705b
I20260808 04:38:30.154227 6558 ts_tablet_manager.cc:1939] T 2d6c909858a4479790f322664f43705b P 0926e782ed534064b69801c09e7d58b0: Deleting consensus metadata
I20260808 04:38:30.156580 6273 catalog_manager.cc:4841] TS 0926e782ed534064b69801c09e7d58b0 (127.15.228.65:34049): tablet 2d6c909858a4479790f322664f43705b (table test-table [id=9057d84288234f97bb0c574860317a11]) successfully deleted
W20260808 04:38:30.239616 6550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:38:30.239876 6550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:38:30.239919 6550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:38:30.244328 6550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:38:30.244410 6550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:38:30.244479 6550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:38:30.249929 6550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:34977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=38889
--tserver_master_addrs=127.15.228.126:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:38:30.251109 6550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:38:30.252496 6550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:38:30.263248 6563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:38:30.263394 6566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:38:30.263278 6564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:38:30.263545 6550 server_base.cc:1034] running on GCE node
I20260808 04:38:30.264034 6550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:38:30.265084 6550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:38:30.266402 6550 hybrid_clock.cc:648] HybridClock initialized: now 1786163910266389 us; error 53 us; skew 500 ppm
Aug 08 04:38:30 dist-test-slave-drl0 krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163910, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:38:30.269457 6550 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:38:30.270416 6550 webserver.cc:460] Webserver started at http://127.15.228.66:38889/ using document root <none> and password file <none>
I20260808 04:38:30.271121 6550 fs_manager.cc:359] Metadata directory not provided
I20260808 04:38:30.271251 6550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:38:30.275251 6550 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:38:30.277697 6576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:30.279393 6550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:38:30.279585 6550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "578f0e1082ca4efebf13e84f1f256ba3"
format_stamp: "Formatted at 2026-08-08 04:38:08 on dist-test-slave-drl0"
server_key: "ed7b3a2d001b66f6bf10fc3d5cc9d9b8"
server_key_iv: "a260d79e5493808627211f8f69e59376"
server_key_version: "encryptionkey@0"
I20260808 04:38:30.280328 6550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:38:30.312983 6550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:38:30.317263 6550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:38:30.317553 6550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:38:30.318457 6550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:38:30.319415 6550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:38:30.319471 6550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:30.319515 6550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:38:30.319531 6550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:30.328967 6550 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:34977
I20260808 04:38:30.329077 6736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:34977 every 8 connection(s)
I20260808 04:38:30.330077 6550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:38:30.332894 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 6550
I20260808 04:38:30.333484 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:40449
--local_ip_for_outbound_sockets=127.15.228.67
--tserver_master_addrs=127.15.228.126:45107
--webserver_port=33263
--webserver_interface=127.15.228.67
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 04:38:30 dist-test-slave-drl0 krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163910, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:38:30.349406 6549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.332243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:41861 (local address 127.15.228.126:45107)
0808 04:38:30.332398 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:30.332402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:30.334103 (+ 1701us) server_negotiation.cc:407] Connection header received
0808 04:38:30.335218 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:30.335224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:30.335304 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:30.335416 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:38:30.337776 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.338442 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.340656 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.340985 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.341131 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:30.344956 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:38:30.344960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:38:30.344963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:38:30.344996 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:38:30.346838 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:30.347534 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:30.347538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:30.347540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:30.347717 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:30.348146 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:30.348159 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:30.348160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:30.348383 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:38:30.348502 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:30.349038 (+ 536us) server_negotiation.cc:299] Negotiation successful
0808 04:38:30.349194 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:38:30.350430 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.332554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.66:41861)
0808 04:38:30.333907 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:30.333967 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 04:38:30.334987 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:38:30.335948 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:38:30.335966 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:38:30.336692 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:38:30.337487 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:30.337516 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.338659 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.338665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:30.340407 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:30.340436 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.341244 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.341253 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:30.341525 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:30.342372 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:38:30.342403 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:38:30.344671 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:38:30.347073 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:38:30.347080 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:38:30.347098 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:38:30.347406 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:38:30.347860 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:38:30.347863 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:38:30.347865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:38:30.348034 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:38:30.348501 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:38:30.348509 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:38:30.348775 (+ 266us) client_negotiation.cc:769] Sending connection context
0808 04:38:30.348980 (+ 205us) client_negotiation.cc:240] Negotiation successful
0808 04:38:30.349340 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1111,"thread_start_us":934,"threads_started":1}
I20260808 04:38:30.351740 6737 heartbeater.cc:347] Connected to a master server at 127.15.228.126:45107
I20260808 04:38:30.352015 6737 heartbeater.cc:464] Registering TS with master...
I20260808 04:38:30.352617 6737 heartbeater.cc:511] Master 127.15.228.126:45107 requested a full tablet report, sending...
I20260808 04:38:30.353818 6295 ts_manager.cc:194] Registered new tserver with Master: 578f0e1082ca4efebf13e84f1f256ba3 (127.15.228.66:34977)
I20260808 04:38:30.354627 6295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:41861
W20260808 04:38:30.507494 6744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:38:30.507876 6744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:38:30.507977 6744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:38:30.512851 6744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:38:30.512921 6744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:38:30.513046 6744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:38:30.518592 6744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:40449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=33263
--tserver_master_addrs=127.15.228.126:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:38:30.519853 6744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:38:30.520959 6744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:38:30.531083 6754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:38:30.531081 6756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:38:30.532277 6744 server_base.cc:1034] running on GCE node
W20260808 04:38:30.535653 6753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:38:30.536197 6744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:38:30.537451 6744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:38:30.538715 6744 hybrid_clock.cc:648] HybridClock initialized: now 1786163910538655 us; error 62 us; skew 500 ppm
Aug 08 04:38:30 dist-test-slave-drl0 krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163910, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:38:30.543005 6744 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:38:30.544112 6744 webserver.cc:460] Webserver started at http://127.15.228.67:33263/ using document root <none> and password file <none>
I20260808 04:38:30.544756 6744 fs_manager.cc:359] Metadata directory not provided
I20260808 04:38:30.544878 6744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:38:30.549386 6744 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260808 04:38:30.552037 6766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:30.553555 6744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 04:38:30.553691 6744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "d389e52d8a484c4891390fbeb7fca81c"
format_stamp: "Formatted at 2026-08-08 04:38:09 on dist-test-slave-drl0"
server_key: "162c24eac55a998ecf211d292d162abf"
server_key_iv: "db6928419f1a614db8917e96b0dea095"
server_key_version: "encryptionkey@0"
I20260808 04:38:30.554255 6744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:38:30.566874 6744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:38:30.570657 6744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:38:30.570883 6744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:38:30.571586 6744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:38:30.572499 6744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:38:30.572547 6744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:30.572614 6744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:38:30.572647 6744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:30.584306 6744 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:40449
I20260808 04:38:30.584342 6918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:40449 every 8 connection(s)
I20260808 04:38:30.585695 6744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:38:30.594363 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 6744
Aug 08 04:38:30 dist-test-slave-drl0 krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163910, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:38:30.606333 6549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.588521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:46383 (local address 127.15.228.126:45107)
0808 04:38:30.588709 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:30.588716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:30.590215 (+ 1499us) server_negotiation.cc:407] Connection header received
0808 04:38:30.591702 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:30.591706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:30.591761 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:30.591852 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:38:30.593763 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.594314 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.596040 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.596450 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.596581 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:30.601455 (+ 4874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:38:30.601461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:38:30.601467 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:38:30.601502 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:38:30.603672 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:30.604469 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:30.604472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:30.604474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:30.604568 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:30.604976 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:30.604979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:30.604980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:30.605163 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:38:30.605269 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:30.605967 (+ 698us) server_negotiation.cc:299] Negotiation successful
0808 04:38:30.606104 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 04:38:30.606731 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.595583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60922 (local address 127.15.228.126:45107)
0808 04:38:30.596045 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:30.596055 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:30.596067 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:38:30.596224 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:30.596229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:30.596293 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:30.596407 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:38:30.597371 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.598282 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.599821 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.600265 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.600405 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:30.602651 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:38:30.602657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:38:30.602661 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:38:30.602713 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:38:30.604775 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:30.605298 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:30.605301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:30.605302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:30.605383 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:30.605722 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:30.605725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:30.605726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:30.605983 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:38:30.606202 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:30.606430 (+ 228us) server_negotiation.cc:299] Negotiation successful
0808 04:38:30.606560 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"spinlock_wait_cycles":2304,"thread_start_us":154,"threads_started":1}
I20260808 04:38:30.607296 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.588951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.67:46383)
0808 04:38:30.589990 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:30.590058 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0808 04:38:30.591384 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:38:30.592101 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:38:30.592112 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:38:30.592718 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:38:30.593543 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:30.593569 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.594501 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.594506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:30.595890 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:30.595904 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.596641 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.596647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:30.596913 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:30.598574 (+ 1661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:38:30.598603 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:38:30.601159 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:38:30.604006 (+ 2847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:38:30.604013 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:38:30.604029 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:38:30.604341 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:38:30.604665 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:38:30.604668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:38:30.604671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:38:30.604820 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:38:30.605287 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:38:30.605296 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:38:30.605649 (+ 353us) client_negotiation.cc:769] Sending connection context
0808 04:38:30.605847 (+ 198us) client_negotiation.cc:240] Negotiation successful
0808 04:38:30.606183 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":567,"threads_started":1}
I20260808 04:38:30.608933 6919 heartbeater.cc:347] Connected to a master server at 127.15.228.126:45107
I20260808 04:38:30.609200 6919 heartbeater.cc:464] Registering TS with master...
I20260808 04:38:30.609839 6919 heartbeater.cc:511] Master 127.15.228.126:45107 requested a full tablet report, sending...
I20260808 04:38:30.611099 6295 ts_manager.cc:194] Registered new tserver with Master: d389e52d8a484c4891390fbeb7fca81c (127.15.228.67:40449)
I20260808 04:38:30.611783 6295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:46383
I20260808 04:38:30.621409 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:38:30.627177 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.622284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60930 (local address 127.15.228.126:45107)
0808 04:38:30.622535 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:30.622541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:30.622738 (+ 197us) server_negotiation.cc:407] Connection header received
0808 04:38:30.623047 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:30.623050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:30.623224 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:30.623370 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:38:30.623738 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.624407 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.625542 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.625898 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.626092 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:30.626550 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:38:30.626703 (+ 153us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 04:38:30.627042 (+ 339us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60930: token expired
Metrics: {"server-negotiator.queue_time_us":108}
W20260808 04:38:30.627259 6924 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60930: token expired
W20260808 04:38:30.627657 6193 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.15.228.126:45107)
I20260808 04:38:30.635944 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.628045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60946 (local address 127.15.228.126:45107)
0808 04:38:30.628172 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:30.628175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:30.628187 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:38:30.628255 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:30.628257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:30.628297 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:30.628361 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:38:30.629043 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.629642 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.630635 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.630864 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.631134 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:30.632623 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:38:30.632626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:38:30.632628 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:38:30.632656 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:38:30.634046 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:30.634503 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:30.634505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:30.634507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:30.634581 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:30.634893 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:30.634896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:30.634897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:30.635147 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:38:30.635269 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:30.635575 (+ 306us) server_negotiation.cc:299] Negotiation successful
0808 04:38:30.635746 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:38:30.638650 6295 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:38:30.643630 6295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:38:30.653553 6453 tablet_service.cc:1467] Processing CreateTablet for tablet 086a45dd8dac4c40948a85fc1ccc30f6 (DEFAULT_TABLE table=test-table [id=82bfd07e339e4d70892a12cd6d55dc69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:38:30.654273 6453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086a45dd8dac4c40948a85fc1ccc30f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:38:30.657734 6937 tablet_bootstrap.cc:492] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0: Bootstrap starting.
I20260808 04:38:30.657909 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.652043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:40449 (local address 127.0.0.1:55020)
0808 04:38:30.652467 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:30.652486 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:38:30.652592 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:38:30.653528 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:38:30.653531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:38:30.653551 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:38:30.653674 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:30.653681 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.655281 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.655285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:30.656245 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:30.656257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.657417 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.657422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:30.657528 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:30.657595 (+ 67us) client_negotiation.cc:769] Sending connection context
0808 04:38:30.657654 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 04:38:30.657734 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":4,"thread_start_us":65,"threads_started":1}
I20260808 04:38:30.658597 6933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.652433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:34977 (local address 127.0.0.1:52384)
0808 04:38:30.653020 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:30.653033 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:38:30.653143 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:38:30.653757 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:38:30.653760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:38:30.653780 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:38:30.653893 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:30.653903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.655435 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.655438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:30.656619 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:30.656630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.658136 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.658140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:30.658240 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:30.658269 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 04:38:30.658379 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 04:38:30.658467 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":25,"thread_start_us":135,"threads_started":1}
I20260808 04:38:30.658665 6932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.652344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.15.228.67:40449)
0808 04:38:30.652935 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:30.652951 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:30.652989 (+ 38us) server_negotiation.cc:407] Connection header received
0808 04:38:30.653082 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:30.653091 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:30.653348 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:30.653533 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:38:30.653944 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.655109 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.656410 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.657203 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.658098 (+ 895us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:30.658232 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:30.658287 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 04:38:30.658499 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":203,"threads_started":1}
I20260808 04:38:30.659215 6937 tablet_bootstrap.cc:654] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0: Neither blocks nor log segments found. Creating new log.
I20260808 04:38:30.659592 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.652633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52384 (local address 127.15.228.66:34977)
0808 04:38:30.652965 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:30.652979 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:30.653123 (+ 144us) server_negotiation.cc:407] Connection header received
0808 04:38:30.653277 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:30.653289 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:30.653505 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:30.653649 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:38:30.654052 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.655308 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.656931 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.657791 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.657897 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:30.658068 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:30.658456 (+ 388us) server_negotiation.cc:299] Negotiation successful
0808 04:38:30.658558 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":127,"threads_started":1}
I20260808 04:38:30.661015 6937 tablet_bootstrap.cc:492] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0: No bootstrap required, opened a new log
I20260808 04:38:30.661132 6937 ts_tablet_manager.cc:1397] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 04:38:30.661526 6937 raft_consensus.cc:348] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:30.661652 6937 raft_consensus.cc:374] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:38:30.661706 6937 raft_consensus.cc:729] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0926e782ed534064b69801c09e7d58b0, State: Initialized, Role: FOLLOWER
I20260808 04:38:30.661849 6937 consensus_queue.cc:260] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:30.662252 6937 ts_tablet_manager.cc:1428] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 04:38:30.662036 6833 tablet_service.cc:1467] Processing CreateTablet for tablet 086a45dd8dac4c40948a85fc1ccc30f6 (DEFAULT_TABLE table=test-table [id=82bfd07e339e4d70892a12cd6d55dc69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:38:30.663729 6833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086a45dd8dac4c40948a85fc1ccc30f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:38:30.666448 6644 tablet_service.cc:1467] Processing CreateTablet for tablet 086a45dd8dac4c40948a85fc1ccc30f6 (DEFAULT_TABLE table=test-table [id=82bfd07e339e4d70892a12cd6d55dc69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:38:30.667490 6644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086a45dd8dac4c40948a85fc1ccc30f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:38:30.672591 6940 tablet_bootstrap.cc:492] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c: Bootstrap starting.
I20260808 04:38:30.674407 6941 tablet_bootstrap.cc:492] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3: Bootstrap starting.
I20260808 04:38:30.674770 6940 tablet_bootstrap.cc:654] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c: Neither blocks nor log segments found. Creating new log.
I20260808 04:38:30.676586 6940 log.cc:824] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c: Log is configured to *not* fsync() on all Append() calls
I20260808 04:38:30.677515 6941 tablet_bootstrap.cc:654] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3: Neither blocks nor log segments found. Creating new log.
I20260808 04:38:30.678445 6941 log.cc:824] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3: Log is configured to *not* fsync() on all Append() calls
I20260808 04:38:30.680629 6940 tablet_bootstrap.cc:492] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c: No bootstrap required, opened a new log
I20260808 04:38:30.681128 6940 ts_tablet_manager.cc:1397] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260808 04:38:30.682381 6941 tablet_bootstrap.cc:492] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3: No bootstrap required, opened a new log
I20260808 04:38:30.682835 6941 ts_tablet_manager.cc:1397] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260808 04:38:30.687832 6940 raft_consensus.cc:348] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:30.688448 6940 raft_consensus.cc:374] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:38:30.688657 6940 raft_consensus.cc:729] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d389e52d8a484c4891390fbeb7fca81c, State: Initialized, Role: FOLLOWER
I20260808 04:38:30.688643 6941 raft_consensus.cc:348] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:30.689113 6941 raft_consensus.cc:374] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:38:30.689229 6941 raft_consensus.cc:729] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 578f0e1082ca4efebf13e84f1f256ba3, State: Initialized, Role: FOLLOWER
I20260808 04:38:30.689669 6940 consensus_queue.cc:260] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:30.689971 6941 consensus_queue.cc:260] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:30.691006 6919 heartbeater.cc:503] Master 127.15.228.126:45107 was elected leader, sending a full tablet report...
I20260808 04:38:30.691273 6737 heartbeater.cc:503] Master 127.15.228.126:45107 was elected leader, sending a full tablet report...
I20260808 04:38:30.691711 6941 ts_tablet_manager.cc:1428] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20260808 04:38:30.693359 6940 ts_tablet_manager.cc:1428] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260808 04:38:30.777776 6939 raft_consensus.cc:482] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:38:30.778014 6939 raft_consensus.cc:504] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:30.779374 6939 leader_election.cc:288] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d389e52d8a484c4891390fbeb7fca81c (127.15.228.67:40449), 578f0e1082ca4efebf13e84f1f256ba3 (127.15.228.66:34977)
I20260808 04:38:30.785100 6949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.779941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:34977 (local address 127.15.228.65:44615)
0808 04:38:30.780578 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:30.780604 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 04:38:30.780780 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:38:30.781238 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:38:30.781242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:38:30.781280 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:38:30.781476 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:30.781485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.782529 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.782535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:30.783526 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:30.783536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.784382 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.784387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:30.784538 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:30.784653 (+ 115us) client_negotiation.cc:769] Sending connection context
0808 04:38:30.784748 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 04:38:30.784873 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":1,"thread_start_us":137,"threads_started":1}
I20260808 04:38:30.785221 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.780196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:44615 (local address 127.15.228.66:34977)
0808 04:38:30.780385 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:30.780391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:30.780900 (+ 509us) server_negotiation.cc:407] Connection header received
0808 04:38:30.781008 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:30.781014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:30.781079 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:30.781168 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:38:30.781787 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.782380 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.783682 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.784202 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.784332 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:30.784388 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:30.784875 (+ 487us) server_negotiation.cc:299] Negotiation successful
0808 04:38:30.785023 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 04:38:30.785372 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.779941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:40449 (local address 127.15.228.65:45865)
0808 04:38:30.780899 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:30.780918 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:38:30.781057 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:38:30.781479 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:38:30.781483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:38:30.781523 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:38:30.781611 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:30.781616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.782676 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.782679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:30.783693 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:30.783705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.784778 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.784786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:30.784915 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:30.784945 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 04:38:30.785043 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 04:38:30.785131 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":176,"thread_start_us":151,"threads_started":1}
I20260808 04:38:30.785372 6932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.780212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:45865 (local address 127.15.228.67:40449)
0808 04:38:30.780396 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:30.780402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:30.781097 (+ 695us) server_negotiation.cc:407] Connection header received
0808 04:38:30.781238 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:30.781242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:30.781302 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:30.781416 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:38:30.781885 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.782460 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.783883 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.784509 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.784656 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:30.784731 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:30.785097 (+ 366us) server_negotiation.cc:299] Negotiation successful
0808 04:38:30.785211 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 04:38:30.786299 6862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086a45dd8dac4c40948a85fc1ccc30f6" candidate_uuid: "0926e782ed534064b69801c09e7d58b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d389e52d8a484c4891390fbeb7fca81c" is_pre_election: true
I20260808 04:38:30.786721 6862 raft_consensus.cc:2454] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0926e782ed534064b69801c09e7d58b0 in term 0.
I20260808 04:38:30.786607 6673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086a45dd8dac4c40948a85fc1ccc30f6" candidate_uuid: "0926e782ed534064b69801c09e7d58b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "578f0e1082ca4efebf13e84f1f256ba3" is_pre_election: true
I20260808 04:38:30.787158 6673 raft_consensus.cc:2454] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0926e782ed534064b69801c09e7d58b0 in term 0.
I20260808 04:38:30.787386 6393 leader_election.cc:302] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0926e782ed534064b69801c09e7d58b0, d389e52d8a484c4891390fbeb7fca81c; no voters:
I20260808 04:38:30.787812 6939 raft_consensus.cc:2781] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:38:30.787956 6939 raft_consensus.cc:482] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:38:30.788028 6939 raft_consensus.cc:3037] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:38:30.789646 6939 raft_consensus.cc:504] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:30.790195 6939 leader_election.cc:288] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 1 election: Requested vote from peers d389e52d8a484c4891390fbeb7fca81c (127.15.228.67:40449), 578f0e1082ca4efebf13e84f1f256ba3 (127.15.228.66:34977)
I20260808 04:38:30.790733 6862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086a45dd8dac4c40948a85fc1ccc30f6" candidate_uuid: "0926e782ed534064b69801c09e7d58b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d389e52d8a484c4891390fbeb7fca81c"
I20260808 04:38:30.790735 6673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086a45dd8dac4c40948a85fc1ccc30f6" candidate_uuid: "0926e782ed534064b69801c09e7d58b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "578f0e1082ca4efebf13e84f1f256ba3"
I20260808 04:38:30.790899 6862 raft_consensus.cc:3037] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:38:30.790899 6673 raft_consensus.cc:3037] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:38:30.792446 6673 raft_consensus.cc:2454] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0926e782ed534064b69801c09e7d58b0 in term 1.
I20260808 04:38:30.792451 6862 raft_consensus.cc:2454] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0926e782ed534064b69801c09e7d58b0 in term 1.
I20260808 04:38:30.792836 6392 leader_election.cc:302] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0926e782ed534064b69801c09e7d58b0, 578f0e1082ca4efebf13e84f1f256ba3; no voters:
I20260808 04:38:30.793257 6939 raft_consensus.cc:2781] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:38:30.793586 6939 raft_consensus.cc:686] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 1 LEADER]: Becoming Leader. State: Replica: 0926e782ed534064b69801c09e7d58b0, State: Running, Role: LEADER
I20260808 04:38:30.793891 6939 consensus_queue.cc:237] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:30.797251 6293 catalog_manager.cc:5495] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0926e782ed534064b69801c09e7d58b0 (127.15.228.65). New cstate: current_term: 1 leader_uuid: "0926e782ed534064b69801c09e7d58b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } health_report { overall_health: HEALTHY } } }
I20260808 04:38:30.828064 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.822704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41274 (local address 127.15.228.65:34049)
0808 04:38:30.823109 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:30.823116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:30.823129 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:38:30.823220 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:30.823222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:30.823280 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:30.823531 (+ 251us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:38:30.823807 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.824352 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.825526 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:30.825754 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:30.825847 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:30.826347 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:38:30.827189 (+ 842us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:38:30.827341 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:30.827515 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 04:38:30.827626 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":166,"threads_started":1}
W20260808 04:38:30.831548 6742 tablet.cc:2369] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:38:30.832684 6673 raft_consensus.cc:1261] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3 [term 1 FOLLOWER]: Refusing update from remote peer 0926e782ed534064b69801c09e7d58b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:38:30.832688 6862 raft_consensus.cc:1261] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c [term 1 FOLLOWER]: Refusing update from remote peer 0926e782ed534064b69801c09e7d58b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:38:30.833458 6939 consensus_queue.cc:1035] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:38:30.833689 6951 consensus_queue.cc:1035] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 04:38:30.836705 6920 tablet.cc:2369] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:38:30.841955 6959 mvcc.cc:205] Tried to move back new op lower bound from 7316127378762600448 to 7316127378614435840. Current Snapshot: MvccSnapshot[applied={T|T < 7316127378762600448}]
I20260808 04:38:30.842499 6956 mvcc.cc:205] Tried to move back new op lower bound from 7316127378762600448 to 7316127378614435840. Current Snapshot: MvccSnapshot[applied={T|T < 7316127378762600448}]
I20260808 04:38:30.852849 6293 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:38:30.853174 6293 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:38:30.857004 6293 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99: Sending DeleteTablet for 3 replicas of tablet 086a45dd8dac4c40948a85fc1ccc30f6
I20260808 04:38:30.858301 6833 tablet_service.cc:1514] Processing DeleteTablet for tablet 086a45dd8dac4c40948a85fc1ccc30f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:38:30 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:55020
I20260808 04:38:30.858381 6644 tablet_service.cc:1514] Processing DeleteTablet for tablet 086a45dd8dac4c40948a85fc1ccc30f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:38:30 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:52384
I20260808 04:38:30.858645 6453 tablet_service.cc:1514] Processing DeleteTablet for tablet 086a45dd8dac4c40948a85fc1ccc30f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:38:30 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:41266
I20260808 04:38:30.859099 6975 tablet_replica.cc:330] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3: stopping tablet replica
I20260808 04:38:30.859108 6973 tablet_replica.cc:330] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c: stopping tablet replica
I20260808 04:38:30.859442 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 6344
I20260808 04:38:30.859510 6973 raft_consensus.cc:2229] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:38:30.859751 6975 raft_consensus.cc:2229] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:38:30.863992 6975 raft_consensus.cc:2258] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:38:30.869022 6975 ts_tablet_manager.cc:1905] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:38:30.871845 6975 ts_tablet_manager.cc:1918] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:38:30.872028 6975 log.cc:1196] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/086a45dd8dac4c40948a85fc1ccc30f6
I20260808 04:38:30.872206 6973 raft_consensus.cc:2258] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:38:30.872510 6975 ts_tablet_manager.cc:1939] T 086a45dd8dac4c40948a85fc1ccc30f6 P 578f0e1082ca4efebf13e84f1f256ba3: Deleting consensus metadata
I20260808 04:38:30.874269 6273 catalog_manager.cc:4841] TS 578f0e1082ca4efebf13e84f1f256ba3 (127.15.228.66:34977): tablet 086a45dd8dac4c40948a85fc1ccc30f6 (table test-table [id=82bfd07e339e4d70892a12cd6d55dc69]) successfully deleted
I20260808 04:38:30.875849 6973 ts_tablet_manager.cc:1905] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:38:30.878532 6973 ts_tablet_manager.cc:1918] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:38:30.878681 6973 log.cc:1196] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/086a45dd8dac4c40948a85fc1ccc30f6
I20260808 04:38:30.879453 6973 ts_tablet_manager.cc:1939] T 086a45dd8dac4c40948a85fc1ccc30f6 P d389e52d8a484c4891390fbeb7fca81c: Deleting consensus metadata
I20260808 04:38:30.881387 6274 catalog_manager.cc:4841] TS d389e52d8a484c4891390fbeb7fca81c (127.15.228.67:40449): tablet 086a45dd8dac4c40948a85fc1ccc30f6 (table test-table [id=82bfd07e339e4d70892a12cd6d55dc69]) successfully deleted
W20260808 04:38:30.883757 6273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.228.65:34049 (error 108)
I20260808 04:38:30.884930 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 6550
I20260808 04:38:30.885174 6933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:30.884672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:34049 (local address 127.0.0.1:41284)
0808 04:38:30.884965 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:30.885067 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.65:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
W20260808 04:38:30.890216 6273 catalog_manager.cc:4584] TS 0926e782ed534064b69801c09e7d58b0 (127.15.228.65:34049): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 086a45dd8dac4c40948a85fc1ccc30f6: Network error: Client connection negotiation failed: client connection to 127.15.228.65:34049: connect: Connection refused (error 111)
I20260808 04:38:30.901598 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 6744
I20260808 04:38:30.909376 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 6240
I20260808 04:38:50.924858 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:45107
--webserver_interface=127.15.228.126
--webserver_port=45783
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:45107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:38:51.060000 6980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:38:51.060323 6980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:38:51.060415 6980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:38:51.064726 6980 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 04:38:51.064792 6980 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 04:38:51.064821 6980 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 04:38:51.064855 6980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:38:51.064898 6980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:38:51.064954 6980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:38:51.064994 6980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:38:51.065030 6980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:38:51.070372 6980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:45107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:45107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=45783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:38:51.071544 6980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:38:51.072638 6980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:38:51.083850 6986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:38:51.084131 6987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:38:51.084052 6980 server_base.cc:1034] running on GCE node
W20260808 04:38:51.084295 6990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:38:51.084982 6980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:38:51.086154 6980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:38:51.087365 6980 hybrid_clock.cc:648] HybridClock initialized: now 1786163931087315 us; error 68 us; skew 500 ppm
Aug 08 04:38:51 dist-test-slave-drl0 krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163931, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:38:51.093379 6980 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:38:51.094430 6980 webserver.cc:460] Webserver started at http://127.15.228.126:45783/ using document root <none> and password file <none>
I20260808 04:38:51.095111 6980 fs_manager.cc:359] Metadata directory not provided
I20260808 04:38:51.095207 6980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:38:51.100512 6980 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:38:51.103252 7003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:51.104714 6980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:38:51.104871 6980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "5025a708d2244712a14fe71c6895fe99"
format_stamp: "Formatted at 2026-08-08 04:38:08 on dist-test-slave-drl0"
server_key: "fd46ca6520eaf2271a69f50cf2ea5979"
server_key_iv: "23ae778879d5bae8503a56739fedf955"
server_key_version: "encryptionkey@0"
I20260808 04:38:51.105480 6980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:38:51.122529 6980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:38:51.126636 6980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:38:51.126878 6980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:38:51.135650 6980 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:45107
I20260808 04:38:51.135875 7078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:45107 every 8 connection(s)
I20260808 04:38:51.137126 6980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:38:51.141696 7079 sys_catalog.cc:263] Verifying existing consensus state
I20260808 04:38:51.143479 7079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99: Bootstrap starting.
I20260808 04:38:51.144896 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 6980
I20260808 04:38:51.145637 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:34049
--local_ip_for_outbound_sockets=127.15.228.65
--tserver_master_addrs=127.15.228.126:45107
--webserver_port=38787
--webserver_interface=127.15.228.65
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:38:51.171350 7079 log.cc:824] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99: Log is configured to *not* fsync() on all Append() calls
I20260808 04:38:51.186533 7079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:38:51.187067 7079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99: Bootstrap complete.
I20260808 04:38:51.193943 7079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5025a708d2244712a14fe71c6895fe99" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45107 } }
I20260808 04:38:51.194485 7079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5025a708d2244712a14fe71c6895fe99, State: Initialized, Role: FOLLOWER
I20260808 04:38:51.195099 7079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5025a708d2244712a14fe71c6895fe99" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45107 } }
I20260808 04:38:51.195281 7079 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:38:51.195353 7079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:38:51.195457 7079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 2 FOLLOWER]: Advancing to term 3
I20260808 04:38:51.198207 7079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5025a708d2244712a14fe71c6895fe99" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45107 } }
I20260808 04:38:51.198763 7079 leader_election.cc:302] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5025a708d2244712a14fe71c6895fe99; no voters:
I20260808 04:38:51.199185 7079 leader_election.cc:288] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 04:38:51.199368 7083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 04:38:51.199645 7083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [term 3 LEADER]: Becoming Leader. State: Replica: 5025a708d2244712a14fe71c6895fe99, State: Running, Role: LEADER
I20260808 04:38:51.199996 7083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5025a708d2244712a14fe71c6895fe99" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45107 } }
I20260808 04:38:51.200517 7079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:38:51.201354 7084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5025a708d2244712a14fe71c6895fe99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5025a708d2244712a14fe71c6895fe99" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45107 } } }
I20260808 04:38:51.201473 7084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [sys.catalog]: This master's current role is: LEADER
I20260808 04:38:51.202428 7085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5025a708d2244712a14fe71c6895fe99. Latest consensus state: current_term: 3 leader_uuid: "5025a708d2244712a14fe71c6895fe99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5025a708d2244712a14fe71c6895fe99" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 45107 } } }
I20260808 04:38:51.202538 7085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99 [sys.catalog]: This master's current role is: LEADER
I20260808 04:38:51.202751 7095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:38:51.208066 7095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:38:51.208791 7095 catalog_manager.cc:1259] Loaded cluster ID: 53b08da430a7494c97ae706acd573c3b
I20260808 04:38:51.208854 7095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:38:51.211550 7095 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:38:51.212311 7095 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99: Loaded TSK: 0
I20260808 04:38:51.212396 7095 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5025a708d2244712a14fe71c6895fe99: Loaded TSK: 1
F20260808 04:38:51.213227 7095 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f25bb5369ad google::LogMessage::Fail() at ??:0
@ 0x7f25bb53988a google::LogMessage::SendToLog() at ??:0
@ 0x7f25bb5364de google::LogMessage::Flush() at ??:0
@ 0x7f25bb537a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f25be1fa18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f25be1f2f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f25c5ea17bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f25c5e7b99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f25c5e7c178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f25c5ebddf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f25c5edc82b std::function<>::operator()() at ??:0
@ 0x7f25c5e7bb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f25c5e7cf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f25c5e79579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f25c5ebd640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f25bd7883d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f25bd788c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f25bd78a3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f25bd779512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f25bedc16db start_thread at ??:0
@ 0x7f25ba03461f clone at ??:0
W20260808 04:38:51.312512 7081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:38:51.312775 7081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:38:51.312870 7081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:38:51.319625 7081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:38:51.319751 7081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:38:51.319856 7081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:38:51.326860 7081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:34049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=38787
--tserver_master_addrs=127.15.228.126:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:38:51.328073 7081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:38:51.329139 7081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:38:51.341104 7142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:38:51.341231 7081 server_base.cc:1034] running on GCE node
W20260808 04:38:51.341104 7145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:38:51.341450 7147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:38:51.341924 7081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:38:51.342646 7081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:38:51.343928 7081 hybrid_clock.cc:648] HybridClock initialized: now 1786163931343903 us; error 43 us; skew 500 ppm
Aug 08 04:38:51 dist-test-slave-drl0 krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163931, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:38:51.346849 7081 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:38:51.347767 7081 webserver.cc:460] Webserver started at http://127.15.228.65:38787/ using document root <none> and password file <none>
I20260808 04:38:51.348351 7081 fs_manager.cc:359] Metadata directory not provided
I20260808 04:38:51.348443 7081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:38:51.352900 7081 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:38:51.355595 7159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:51.357517 7081 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:38:51.357668 7081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "0926e782ed534064b69801c09e7d58b0"
format_stamp: "Formatted at 2026-08-08 04:38:08 on dist-test-slave-drl0"
server_key: "9443282b05baf9d7342ee18065d52801"
server_key_iv: "b8d1aca71b308b768cc575a7c1d22c1f"
server_key_version: "encryptionkey@0"
I20260808 04:38:51.358242 7081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:38:51.379452 7081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:38:51.383189 7081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:38:51.383410 7081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:38:51.384048 7081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:38:51.385375 7167 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 04:38:51.391208 7081 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 04:38:51.391372 7081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260808 04:38:51.391492 7081 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 04:38:51.394484 7081 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 04:38:51.394572 7081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260808 04:38:51.394886 7167 tablet_bootstrap.cc:492] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0: Bootstrap starting.
I20260808 04:38:51.406852 7081 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:34049
I20260808 04:38:51.407140 7311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:34049 every 8 connection(s)
I20260808 04:38:51.407990 7081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:38:51.418339 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 7081
I20260808 04:38:51.418890 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:34977
--local_ip_for_outbound_sockets=127.15.228.66
--tserver_master_addrs=127.15.228.126:45107
--webserver_port=38889
--webserver_interface=127.15.228.66
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:38:51.420044 7167 log.cc:824] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0: Log is configured to *not* fsync() on all Append() calls
I20260808 04:38:51.427697 7167 tablet_bootstrap.cc:492] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 04:38:51.428087 7167 tablet_bootstrap.cc:492] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0: Bootstrap complete.
I20260808 04:38:51.429325 7167 ts_tablet_manager.cc:1397] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0: Time spent bootstrapping tablet: real 0.035s user 0.018s sys 0.014s
I20260808 04:38:51.433466 7167 raft_consensus.cc:348] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:51.434031 7167 raft_consensus.cc:729] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0926e782ed534064b69801c09e7d58b0, State: Initialized, Role: FOLLOWER
I20260808 04:38:51.434674 7167 consensus_queue.cc:260] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:51.436086 7167 ts_tablet_manager.cc:1428] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
W20260808 04:38:51.503144 7317 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:38:51.410178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.65:46039)
0808 04:38:51.411400 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:51.411513 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0808 04:38:51.412664 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:38:51.502269 (+ 89605us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":287,"threads_started":1}
W20260808 04:38:51.503657 7312 heartbeater.cc:644] Failed to heartbeat to 127.15.228.126:45107 (0 consecutive failures): Network error: Failed to ping master at 127.15.228.126:45107: Client connection negotiation failed: client connection to 127.15.228.126:45107: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 04:38:51.504532 7317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:51.504188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.65:54675)
0808 04:38:51.504346 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:51.504415 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260808 04:38:51.505182 7317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:51.504980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.65:34895)
0808 04:38:51.505083 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:51.505137 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260808 04:38:51.505349 7312 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:38:51.573204 7319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:38:51.573655 7319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:38:51.573736 7319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:38:51.582545 7319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:38:51.582659 7319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:38:51.582800 7319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:38:51.594424 7319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:34977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=38889
--tserver_master_addrs=127.15.228.126:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:38:51.595953 7319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:38:51.597509 7319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:38:51.613333 7337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:38:51.613416 7333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:38:51.613513 7319 server_base.cc:1034] running on GCE node
W20260808 04:38:51.613703 7334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:38:51.614158 7319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:38:51.615021 7319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:38:51.616255 7319 hybrid_clock.cc:648] HybridClock initialized: now 1786163931616216 us; error 57 us; skew 500 ppm
Aug 08 04:38:51 dist-test-slave-drl0 krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163931, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:38:51.619252 7319 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:38:51.620172 7319 webserver.cc:460] Webserver started at http://127.15.228.66:38889/ using document root <none> and password file <none>
I20260808 04:38:51.620754 7319 fs_manager.cc:359] Metadata directory not provided
I20260808 04:38:51.620844 7319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:38:51.625118 7319 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:38:51.627384 7345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:51.628688 7319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:38:51.628845 7319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "578f0e1082ca4efebf13e84f1f256ba3"
format_stamp: "Formatted at 2026-08-08 04:38:08 on dist-test-slave-drl0"
server_key: "ed7b3a2d001b66f6bf10fc3d5cc9d9b8"
server_key_iv: "a260d79e5493808627211f8f69e59376"
server_key_version: "encryptionkey@0"
I20260808 04:38:51.629406 7319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:38:51.641158 7319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:38:51.644446 7319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:38:51.644657 7319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:38:51.645270 7319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:38:51.646118 7319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:38:51.646188 7319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:51.646265 7319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:38:51.646307 7319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:51.657006 7319 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:34977
I20260808 04:38:51.657078 7499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:34977 every 8 connection(s)
I20260808 04:38:51.658061 7319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:38:51.660360 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 7319
I20260808 04:38:51.660929 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:40449
--local_ip_for_outbound_sockets=127.15.228.67
--tserver_master_addrs=127.15.228.126:45107
--webserver_port=33263
--webserver_interface=127.15.228.67
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:38:51.663419 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:51.660401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.66:35087)
0808 04:38:51.660981 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:51.661160 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":155,"threads_started":1}
W20260808 04:38:51.664345 7501 heartbeater.cc:644] Failed to heartbeat to 127.15.228.126:45107 (0 consecutive failures): Network error: Failed to ping master at 127.15.228.126:45107: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
I20260808 04:38:51.665354 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:51.664989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.66:45861)
0808 04:38:51.665179 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:51.665258 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 04:38:51.666512 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:51.666077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.66:48005)
0808 04:38:51.666279 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:51.666378 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260808 04:38:51.666920 7501 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 04:38:51.793936 7505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:38:51.794430 7505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:38:51.794536 7505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:38:51.799156 7505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:38:51.799227 7505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:38:51.799292 7505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:38:51.805083 7505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:40449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=33263
--tserver_master_addrs=127.15.228.126:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:38:51.806329 7505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:38:51.807477 7505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:38:51.817708 7513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:38:51.817727 7515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:38:51.817750 7517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:38:51.817960 7505 server_base.cc:1034] running on GCE node
I20260808 04:38:51.818380 7505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:38:51.819017 7505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:38:51.820192 7505 hybrid_clock.cc:648] HybridClock initialized: now 1786163931820175 us; error 32 us; skew 500 ppm
Aug 08 04:38:51 dist-test-slave-drl0 krb5kdc[5477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163931, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:38:51.823406 7505 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:38:51.824285 7505 webserver.cc:460] Webserver started at http://127.15.228.67:33263/ using document root <none> and password file <none>
I20260808 04:38:51.824896 7505 fs_manager.cc:359] Metadata directory not provided
I20260808 04:38:51.824986 7505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:38:51.829087 7505 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:38:51.831854 7525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:51.833552 7505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:38:51.833694 7505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "d389e52d8a484c4891390fbeb7fca81c"
format_stamp: "Formatted at 2026-08-08 04:38:09 on dist-test-slave-drl0"
server_key: "162c24eac55a998ecf211d292d162abf"
server_key_iv: "db6928419f1a614db8917e96b0dea095"
server_key_version: "encryptionkey@0"
I20260808 04:38:51.834270 7505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:38:51.853988 7505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:38:51.858032 7505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:38:51.858268 7505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:38:51.858919 7505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:38:51.859851 7505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:38:51.859920 7505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:51.860016 7505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:38:51.860064 7505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:38:51.870235 7505 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:40449
I20260808 04:38:51.870265 7690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:40449 every 8 connection(s)
I20260808 04:38:51.871361 7505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:38:51.875397 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:51.873485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.67:46209)
0808 04:38:51.874001 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:51.874109 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":118,"threads_started":1}
W20260808 04:38:51.876194 7691 heartbeater.cc:644] Failed to heartbeat to 127.15.228.126:45107 (0 consecutive failures): Network error: Failed to ping master at 127.15.228.126:45107: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
I20260808 04:38:51.877166 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:51.876828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.67:47361)
0808 04:38:51.876969 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:51.877024 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 04:38:51.877948 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:51.877708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.67:51067)
0808 04:38:51.877851 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:51.877900 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260808 04:38:51.878213 7691 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 04:38:51.881567 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 7505
I20260808 04:38:52.506882 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:52.506240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.65:44221)
0808 04:38:52.506678 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:52.506765 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":115,"threads_started":1}
I20260808 04:38:52.668383 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:52.667795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.66:57267)
0808 04:38:52.668187 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:52.668258 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":117,"threads_started":1}
I20260808 04:38:52.879503 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:52.879025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.67:48235)
0808 04:38:52.879338 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:52.879399 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":91,"threads_started":1}
I20260808 04:38:53.070055 7701 raft_consensus.cc:482] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:38:53.070279 7701 raft_consensus.cc:504] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:53.071494 7701 leader_election.cc:288] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d389e52d8a484c4891390fbeb7fca81c (127.15.228.67:40449), 578f0e1082ca4efebf13e84f1f256ba3 (127.15.228.66:34977)
Aug 08 04:38:53 dist-test-slave-drl0 krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163931, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.66@KRBTEST.COM
Aug 08 04:38:53 dist-test-slave-drl0 krb5kdc[5477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163931, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.67@KRBTEST.COM
I20260808 04:38:53.088194 7705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:53.072445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:59705 (local address 127.15.228.67:40449)
0808 04:38:53.072796 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:53.072816 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:53.072854 (+ 38us) server_negotiation.cc:407] Connection header received
0808 04:38:53.073810 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:53.073819 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:53.074182 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:53.074604 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:38:53.076373 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:53.077746 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:53.079222 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:53.079952 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:53.080133 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:53.083126 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:38:53.083131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:38:53.083141 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:38:53.083171 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:38:53.085003 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:53.086020 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:53.086026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:53.086030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:53.086193 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:53.086537 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:53.086546 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:53.086548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:53.087032 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:38:53.087171 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:53.087738 (+ 567us) server_negotiation.cc:299] Negotiation successful
0808 04:38:53.088017 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":111,"threads_started":1}
I20260808 04:38:53.088176 7702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:53.071931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:34977 (local address 127.15.228.65:50695)
0808 04:38:53.072379 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:53.072396 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:38:53.072532 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:38:53.074751 (+ 2219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:38:53.074760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:38:53.075474 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:38:53.076111 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:53.076130 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:53.077689 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:53.077693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:53.078854 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:53.078869 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:53.079773 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:53.079777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:53.079972 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:53.080699 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:38:53.080722 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:38:53.082536 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:38:53.084978 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:38:53.084985 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:38:53.084999 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:38:53.085397 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:38:53.086228 (+ 831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:38:53.086234 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:38:53.086239 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:38:53.086425 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:38:53.087165 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:38:53.087168 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:38:53.087421 (+ 253us) client_negotiation.cc:769] Sending connection context
0808 04:38:53.087607 (+ 186us) client_negotiation.cc:240] Negotiation successful
0808 04:38:53.087958 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":113,"spinlock_wait_cycles":13056,"thread_start_us":108,"threads_started":1}
I20260808 04:38:53.088333 7704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:53.071931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:40449 (local address 127.15.228.65:59705)
0808 04:38:53.072529 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:53.072545 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:38:53.072659 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:38:53.074751 (+ 2092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:38:53.074760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:38:53.075469 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:38:53.076108 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:53.076129 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:53.077951 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:53.077954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:53.079044 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:38:53.079059 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:38:53.080089 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:38:53.080092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:38:53.080190 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:53.081061 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:38:53.081080 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:38:53.082920 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:38:53.085148 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:38:53.085160 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:38:53.085161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:38:53.085809 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:38:53.086332 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:38:53.086335 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:38:53.086336 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:38:53.086427 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:38:53.087152 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:38:53.087159 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:38:53.087421 (+ 262us) client_negotiation.cc:769] Sending connection context
0808 04:38:53.087606 (+ 185us) client_negotiation.cc:240] Negotiation successful
0808 04:38:53.088099 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":326,"spinlock_wait_cycles":29184,"thread_start_us":73,"threads_started":1}
I20260808 04:38:53.089340 7703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:53.072101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:50695 (local address 127.15.228.66:34977)
0808 04:38:53.072507 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0808 04:38:53.072541 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0808 04:38:53.072595 (+ 54us) server_negotiation.cc:407] Connection header received
0808 04:38:53.073602 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:38:53.073617 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:38:53.074164 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:38:53.074604 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:38:53.076359 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:53.077474 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:53.079057 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:38:53.079589 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:38:53.079793 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:38:53.082848 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:38:53.082858 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:38:53.082878 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:38:53.082910 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:38:53.084847 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:53.085718 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:53.085731 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:53.085741 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:53.086106 (+ 365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:38:53.086533 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:38:53.086540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:38:53.086542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:38:53.087040 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:38:53.087184 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 04:38:53.088005 (+ 821us) server_negotiation.cc:299] Negotiation successful
0808 04:38:53.089192 (+ 1187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":85,"threads_started":1}
W20260808 04:38:53.091593 7163 leader_election.cc:341] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d389e52d8a484c4891390fbeb7fca81c (127.15.228.67:40449): Not found: Tablet not found: 086a45dd8dac4c40948a85fc1ccc30f6
W20260808 04:38:53.091845 7162 leader_election.cc:341] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 578f0e1082ca4efebf13e84f1f256ba3 (127.15.228.66:34977): Not found: Tablet not found: 086a45dd8dac4c40948a85fc1ccc30f6
I20260808 04:38:53.091984 7162 leader_election.cc:302] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0926e782ed534064b69801c09e7d58b0; no voters: 578f0e1082ca4efebf13e84f1f256ba3, d389e52d8a484c4891390fbeb7fca81c
I20260808 04:38:53.092242 7701 raft_consensus.cc:2726] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:38:53.508427 7702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:53.508087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.65:53209)
0808 04:38:53.508269 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:53.508337 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260808 04:38:53.669826 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:53.669359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.66:48889)
0808 04:38:53.669661 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:53.669726 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":82,"threads_started":1}
I20260808 04:38:53.880728 7708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:53.880283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.67:59589)
0808 04:38:53.880578 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:53.880637 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":93,"threads_started":1}
I20260808 04:38:54.509976 7709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:54.509470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.65:59225)
0808 04:38:54.509817 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:54.509877 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":122,"threads_started":1}
I20260808 04:38:54.671304 7710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:54.670790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.66:57855)
0808 04:38:54.671125 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:54.671186 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":101,"threads_started":1}
I20260808 04:38:54.882206 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:54.881732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.67:54727)
0808 04:38:54.882052 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:54.882109 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":80,"threads_started":1}
I20260808 04:38:55.067710 7712 raft_consensus.cc:482] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:38:55.067914 7712 raft_consensus.cc:504] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:55.068430 7712 leader_election.cc:288] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d389e52d8a484c4891390fbeb7fca81c (127.15.228.67:40449), 578f0e1082ca4efebf13e84f1f256ba3 (127.15.228.66:34977)
W20260808 04:38:55.069535 7162 leader_election.cc:341] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 578f0e1082ca4efebf13e84f1f256ba3 (127.15.228.66:34977): Not found: Tablet not found: 086a45dd8dac4c40948a85fc1ccc30f6
W20260808 04:38:55.069658 7163 leader_election.cc:341] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d389e52d8a484c4891390fbeb7fca81c (127.15.228.67:40449): Not found: Tablet not found: 086a45dd8dac4c40948a85fc1ccc30f6
I20260808 04:38:55.069829 7163 leader_election.cc:302] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0926e782ed534064b69801c09e7d58b0; no voters: 578f0e1082ca4efebf13e84f1f256ba3, d389e52d8a484c4891390fbeb7fca81c
I20260808 04:38:55.069993 7712 raft_consensus.cc:2726] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:38:55.512073 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:55.511419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.65:46317)
0808 04:38:55.511869 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:55.511946 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":87,"threads_started":1}
I20260808 04:38:55.672905 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:55.672446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.66:38243)
0808 04:38:55.672732 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:55.672802 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":83,"threads_started":1}
I20260808 04:38:55.883963 7715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:55.883399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.67:36793)
0808 04:38:55.883796 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:55.883875 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":211,"threads_started":1}
I20260808 04:38:56.513842 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:56.513111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.65:37469)
0808 04:38:56.513602 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:56.513700 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":88,"threads_started":1}
I20260808 04:38:56.674261 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:56.673827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.66:49343)
0808 04:38:56.674099 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:56.674160 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260808 04:38:56.885392 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:56.884933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.67:49573)
0808 04:38:56.885221 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:56.885285 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":89,"threads_started":1}
I20260808 04:38:57.515735 7721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:57.515189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.65:35625)
0808 04:38:57.515526 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:57.515601 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":83,"threads_started":1}
I20260808 04:38:57.676162 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:57.675336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.66:33293)
0808 04:38:57.675705 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:57.675991 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":81,"threads_started":1}
I20260808 04:38:57.786497 7725 raft_consensus.cc:482] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:38:57.786650 7725 raft_consensus.cc:504] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:38:57.787227 7725 leader_election.cc:288] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d389e52d8a484c4891390fbeb7fca81c (127.15.228.67:40449), 578f0e1082ca4efebf13e84f1f256ba3 (127.15.228.66:34977)
W20260808 04:38:57.788030 7163 leader_election.cc:341] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d389e52d8a484c4891390fbeb7fca81c (127.15.228.67:40449): Not found: Tablet not found: 086a45dd8dac4c40948a85fc1ccc30f6
W20260808 04:38:57.788130 7162 leader_election.cc:341] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 578f0e1082ca4efebf13e84f1f256ba3 (127.15.228.66:34977): Not found: Tablet not found: 086a45dd8dac4c40948a85fc1ccc30f6
I20260808 04:38:57.788241 7162 leader_election.cc:302] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0926e782ed534064b69801c09e7d58b0; no voters: 578f0e1082ca4efebf13e84f1f256ba3, d389e52d8a484c4891390fbeb7fca81c
I20260808 04:38:57.788403 7725 raft_consensus.cc:2726] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:38:57.886754 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:57.886282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.67:41411)
0808 04:38:57.886597 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:57.886655 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
I20260808 04:38:58.517237 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:58.516691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.65:40499)
0808 04:38:58.517036 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:58.517103 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
I20260808 04:38:58.678177 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:58.677431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.66:49011)
0808 04:38:58.677939 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:58.678052 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":110,"threads_started":1}
I20260808 04:38:58.888528 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:58.887807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.67:46143)
0808 04:38:58.888355 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:58.888422 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":242,"threads_started":1}
I20260808 04:38:59.519048 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:59.518496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.65:57935)
0808 04:38:59.518802 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:59.518868 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":90,"threads_started":1}
I20260808 04:38:59.680272 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:59.679512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.66:44145)
0808 04:38:59.679892 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:59.679987 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":119,"threads_started":1}
I20260808 04:38:59.890165 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:38:59.889404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.67:55317)
0808 04:38:59.889871 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0808 04:38:59.889970 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":113,"threads_started":1}
I20260808 04:39:00.520819 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:00.520357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.65:55455)
0808 04:39:00.520646 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:00.520707 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":86,"threads_started":1}
I20260808 04:39:00.681984 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:00.681213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.66:33457)
0808 04:39:00.681666 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:00.681772 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":84,"threads_started":1}
I20260808 04:39:00.891739 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:00.891261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.67:59747)
0808 04:39:00.891570 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:00.891634 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":90,"threads_started":1}
I20260808 04:39:01.522316 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:01.521870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.65:35347)
0808 04:39:01.522152 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:01.522210 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":80,"threads_started":1}
I20260808 04:39:01.684221 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:01.683532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.66:46401)
0808 04:39:01.683940 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:01.684032 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":102,"threads_started":1}
I20260808 04:39:01.859190 7742 raft_consensus.cc:482] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:39:01.859426 7742 raft_consensus.cc:504] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:39:01.860088 7742 leader_election.cc:288] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d389e52d8a484c4891390fbeb7fca81c (127.15.228.67:40449), 578f0e1082ca4efebf13e84f1f256ba3 (127.15.228.66:34977)
W20260808 04:39:01.861542 7162 leader_election.cc:341] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 578f0e1082ca4efebf13e84f1f256ba3 (127.15.228.66:34977): Not found: Tablet not found: 086a45dd8dac4c40948a85fc1ccc30f6
W20260808 04:39:01.861778 7163 leader_election.cc:341] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d389e52d8a484c4891390fbeb7fca81c (127.15.228.67:40449): Not found: Tablet not found: 086a45dd8dac4c40948a85fc1ccc30f6
I20260808 04:39:01.861919 7163 leader_election.cc:302] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0926e782ed534064b69801c09e7d58b0; no voters: 578f0e1082ca4efebf13e84f1f256ba3, d389e52d8a484c4891390fbeb7fca81c
I20260808 04:39:01.862293 7742 raft_consensus.cc:2726] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:39:01.893247 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:01.892788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.67:38039)
0808 04:39:01.893079 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:01.893143 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":83,"threads_started":1}
I20260808 04:39:02.523733 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:02.523227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.65:34657)
0808 04:39:02.523558 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:02.523631 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":86,"threads_started":1}
I20260808 04:39:02.685702 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:02.684989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.66:39569)
0808 04:39:02.685460 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:02.685560 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":101,"threads_started":1}
I20260808 04:39:02.894802 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:02.894164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.67:39289)
0808 04:39:02.894624 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:02.894695 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":85,"threads_started":1}
I20260808 04:39:03.525082 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:03.524620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.65:46353)
0808 04:39:03.524919 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:03.524981 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":103,"threads_started":1}
I20260808 04:39:03.687294 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:03.686741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.66:33897)
0808 04:39:03.687127 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:03.687189 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":90,"threads_started":1}
I20260808 04:39:03.896140 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:03.895656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.67:39307)
0808 04:39:03.895972 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:03.896032 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":107,"threads_started":1}
I20260808 04:39:04.526520 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:04.526026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.65:36471)
0808 04:39:04.526343 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:04.526403 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":100,"threads_started":1}
I20260808 04:39:04.688654 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:04.688185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.66:51791)
0808 04:39:04.688501 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:04.688556 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":100,"threads_started":1}
I20260808 04:39:04.897491 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:04.897008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.67:40077)
0808 04:39:04.897307 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:04.897368 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":80,"threads_started":1}
I20260808 04:39:05.246261 7763 raft_consensus.cc:482] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:39:05.246457 7763 raft_consensus.cc:504] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389e52d8a484c4891390fbeb7fca81c" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 40449 } } peers { permanent_uuid: "578f0e1082ca4efebf13e84f1f256ba3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 34977 } } peers { permanent_uuid: "0926e782ed534064b69801c09e7d58b0" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 34049 } }
I20260808 04:39:05.247105 7763 leader_election.cc:288] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d389e52d8a484c4891390fbeb7fca81c (127.15.228.67:40449), 578f0e1082ca4efebf13e84f1f256ba3 (127.15.228.66:34977)
W20260808 04:39:05.248219 7163 leader_election.cc:341] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d389e52d8a484c4891390fbeb7fca81c (127.15.228.67:40449): Not found: Tablet not found: 086a45dd8dac4c40948a85fc1ccc30f6
W20260808 04:39:05.248449 7162 leader_election.cc:341] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 578f0e1082ca4efebf13e84f1f256ba3 (127.15.228.66:34977): Not found: Tablet not found: 086a45dd8dac4c40948a85fc1ccc30f6
I20260808 04:39:05.248581 7162 leader_election.cc:302] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0926e782ed534064b69801c09e7d58b0; no voters: 578f0e1082ca4efebf13e84f1f256ba3, d389e52d8a484c4891390fbeb7fca81c
I20260808 04:39:05.248778 7763 raft_consensus.cc:2726] T 086a45dd8dac4c40948a85fc1ccc30f6 P 0926e782ed534064b69801c09e7d58b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 04:39:05.527993 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:05.527485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.65:46463)
0808 04:39:05.527799 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:05.527881 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":85,"threads_started":1}
I20260808 04:39:05.689977 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:05.689444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.66:49093)
0808 04:39:05.689728 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:05.689868 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":80,"threads_started":1}
I20260808 04:39:05.899267 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:05.898620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.67:43199)
0808 04:39:05.899045 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:05.899134 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":162,"threads_started":1}
I20260808 04:39:06.529645 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:06.529208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.65:36703)
0808 04:39:06.529490 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:06.529548 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":102,"threads_started":1}
I20260808 04:39:06.691859 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:06.691187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.66:57381)
0808 04:39:06.691621 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:06.691719 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":86,"threads_started":1}
I20260808 04:39:06.890033 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 7081
I20260808 04:39:06.901522 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:06.900828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:45107 (local address 127.15.228.67:58573)
0808 04:39:06.901315 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:06.901402 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.126:45107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":145,"threads_started":1}
I20260808 04:39:06.907074 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 7319
I20260808 04:39:06.922228 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 7505
2026-08-08T04:39:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (60272 ms)
[----------] 30 tests from SecurityITest (277971 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:07 dist-test-slave-drl0 krb5kdc[7774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:07 dist-test-slave-drl0 krb5kdc[7774](info): set up 2 sockets
Aug 08 04:39:07 dist-test-slave-drl0 krb5kdc[7774](info): commencing operation
krb5kdc: starting...
W20260808 04:39:08.222985 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.268s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:08 dist-test-slave-drl0 krb5kdc[7774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:39:08.376354 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:43567
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:43567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:08.510680 7794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:08.511031 7794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:08.511142 7794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:08.515245 7794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:08.515321 7794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:08.515357 7794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:08.515414 7794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:08.515451 7794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:08.520699 7794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:43567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:43567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:08.521921 7794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:08.523005 7794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:08.532419 7806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:08.532523 7794 server_base.cc:1034] running on GCE node
W20260808 04:39:08.532469 7808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:08.532451 7805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:08.533608 7794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:08.534847 7794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:08.536167 7794 hybrid_clock.cc:648] HybridClock initialized: now 1786163948536113 us; error 68 us; skew 500 ppm
Aug 08 04:39:08 dist-test-slave-drl0 krb5kdc[7774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163948, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:08.539244 7794 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:39:08.540158 7794 webserver.cc:460] Webserver started at http://127.15.228.126:34259/ using document root <none> and password file <none>
I20260808 04:39:08.540742 7794 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:08.540835 7794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:08.541095 7794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:08.543658 7794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "fef141b050654dbc82b0f7d0e5e6a832"
format_stamp: "Formatted at 2026-08-08 04:39:08 on dist-test-slave-drl0"
server_key: "f7b36d145d427e1cb867c4b55ed331d6"
server_key_iv: "fe15c98924d1430d07bd3489e5e15cdd"
server_key_version: "encryptionkey@0"
I20260808 04:39:08.544549 7794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "fef141b050654dbc82b0f7d0e5e6a832"
format_stamp: "Formatted at 2026-08-08 04:39:08 on dist-test-slave-drl0"
server_key: "f7b36d145d427e1cb867c4b55ed331d6"
server_key_iv: "fe15c98924d1430d07bd3489e5e15cdd"
server_key_version: "encryptionkey@0"
I20260808 04:39:08.548775 7794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:39:08.551656 7820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:08.553493 7794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:39:08.553793 7794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "fef141b050654dbc82b0f7d0e5e6a832"
format_stamp: "Formatted at 2026-08-08 04:39:08 on dist-test-slave-drl0"
server_key: "f7b36d145d427e1cb867c4b55ed331d6"
server_key_iv: "fe15c98924d1430d07bd3489e5e15cdd"
server_key_version: "encryptionkey@0"
I20260808 04:39:08.553988 7794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:08.584954 7794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:08.588871 7794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:08.589119 7794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:08.598739 7794 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:43567
I20260808 04:39:08.598748 7888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:43567 every 8 connection(s)
I20260808 04:39:08.599891 7794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:08.603044 7890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:08.606900 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 7794
I20260808 04:39:08.607122 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:08.607430 16273 external_mini_cluster.cc:1428] Setting key dd99473e77685436924dee9f74f91bfc
I20260808 04:39:08.609467 7890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832: Bootstrap starting.
I20260808 04:39:08.612016 7890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:08.612927 7890 log.cc:824] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:08.616181 7890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832: No bootstrap required, opened a new log
Aug 08 04:39:08 dist-test-slave-drl0 krb5kdc[7774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:39:08.622241 7890 raft_consensus.cc:348] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef141b050654dbc82b0f7d0e5e6a832" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43567 } }
I20260808 04:39:08.622560 7890 raft_consensus.cc:374] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:08.622613 7890 raft_consensus.cc:729] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef141b050654dbc82b0f7d0e5e6a832, State: Initialized, Role: FOLLOWER
I20260808 04:39:08.623184 7890 consensus_queue.cc:260] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef141b050654dbc82b0f7d0e5e6a832" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43567 } }
I20260808 04:39:08.623358 7890 raft_consensus.cc:388] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:08.623450 7890 raft_consensus.cc:482] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:08.623555 7890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:08.624744 7890 raft_consensus.cc:504] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef141b050654dbc82b0f7d0e5e6a832" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43567 } }
I20260808 04:39:08.625133 7890 leader_election.cc:302] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fef141b050654dbc82b0f7d0e5e6a832; no voters:
I20260808 04:39:08.625535 7890 leader_election.cc:288] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:08.625660 7896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:08.626000 7896 raft_consensus.cc:686] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832 [term 1 LEADER]: Becoming Leader. State: Replica: fef141b050654dbc82b0f7d0e5e6a832, State: Running, Role: LEADER
I20260808 04:39:08.626369 7896 consensus_queue.cc:237] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef141b050654dbc82b0f7d0e5e6a832" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43567 } }
I20260808 04:39:08.626694 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:08.609369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58988 (local address 127.15.228.126:43567)
0808 04:39:08.610104 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:08.610128 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:08.610181 (+ 53us) server_negotiation.cc:407] Connection header received
0808 04:39:08.610874 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:08.610891 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:08.611362 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:08.611662 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:08.612671 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:08.613979 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:08.615503 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:08.616139 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:08.616280 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:08.619820 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:08.619829 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:08.619854 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:08.619902 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:08.623026 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:08.623633 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:08.623641 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:08.623648 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:08.623801 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:08.624302 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:08.624306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:08.624308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:08.624899 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:08.625103 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:08.625398 (+ 295us) server_negotiation.cc:299] Negotiation successful
0808 04:39:08.625720 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":161,"threads_started":1}
I20260808 04:39:08.626974 7890 sys_catalog.cc:564] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:08.629019 7899 sys_catalog.cc:455] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fef141b050654dbc82b0f7d0e5e6a832. Latest consensus state: current_term: 1 leader_uuid: "fef141b050654dbc82b0f7d0e5e6a832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef141b050654dbc82b0f7d0e5e6a832" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43567 } } }
I20260808 04:39:08.629174 7899 sys_catalog.cc:458] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:08.629599 7897 sys_catalog.cc:455] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fef141b050654dbc82b0f7d0e5e6a832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef141b050654dbc82b0f7d0e5e6a832" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43567 } } }
I20260808 04:39:08.629690 7897 sys_catalog.cc:458] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:08.630484 7915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:08.633428 7915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:08.640723 7915 catalog_manager.cc:1347] Generated new cluster ID: 8159aceb65084d7e93a8b8a1922e1aa2
I20260808 04:39:08.640797 7915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:08.650925 7915 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:08.652189 7915 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:08.671195 7915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832: Generated new TSK 0
I20260808 04:39:08.672796 7915 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
I20260808 04:39:08.711249 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:43567
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:08.846055 7925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:08.846379 7925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:08.846431 7925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:08.851697 7925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:08.851778 7925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:08.851848 7925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:39:08.857623 7925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:43567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:08.858958 7925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:08.860215 7925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:08.872601 7930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:08.872601 7933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:08.872772 7925 server_base.cc:1034] running on GCE node
W20260808 04:39:08.872601 7935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:08.873579 7925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:08.874312 7925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:08.875510 7925 hybrid_clock.cc:648] HybridClock initialized: now 1786163948875480 us; error 48 us; skew 500 ppm
Aug 08 04:39:08 dist-test-slave-drl0 krb5kdc[7774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163948, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:08.878660 7925 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:39:08.879681 7925 webserver.cc:460] Webserver started at http://127.15.228.65:35331/ using document root <none> and password file <none>
I20260808 04:39:08.880293 7925 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:08.880398 7925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:08.880642 7925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:08.883922 7925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "75f42f3d6cff4a8e86f48a9339f810de"
format_stamp: "Formatted at 2026-08-08 04:39:08 on dist-test-slave-drl0"
server_key: "003f9bda7e29427c9aadb3910025779e"
server_key_iv: "8f2e1f91296994e81b2a3915c9b7a069"
server_key_version: "encryptionkey@0"
I20260808 04:39:08.884929 7925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "75f42f3d6cff4a8e86f48a9339f810de"
format_stamp: "Formatted at 2026-08-08 04:39:08 on dist-test-slave-drl0"
server_key: "003f9bda7e29427c9aadb3910025779e"
server_key_iv: "8f2e1f91296994e81b2a3915c9b7a069"
server_key_version: "encryptionkey@0"
I20260808 04:39:08.889600 7925 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:39:08.893268 7942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:08.894848 7925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:39:08.895135 7925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "75f42f3d6cff4a8e86f48a9339f810de"
format_stamp: "Formatted at 2026-08-08 04:39:08 on dist-test-slave-drl0"
server_key: "003f9bda7e29427c9aadb3910025779e"
server_key_iv: "8f2e1f91296994e81b2a3915c9b7a069"
server_key_version: "encryptionkey@0"
I20260808 04:39:08.895287 7925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:08.909016 7925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:08.912617 7925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:08.912853 7925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:08.913529 7925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:39:08.914767 7925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:39:08.914913 7925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:08.915102 7925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:39:08.915158 7925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:08.930408 7925 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:44387
I20260808 04:39:08.930590 8093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:44387 every 8 connection(s)
I20260808 04:39:08.931838 7925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:39:08.941146 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 7925
I20260808 04:39:08.941347 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:39:08.941746 16273 external_mini_cluster.cc:1428] Setting key 2a15b1f054036856b08799bb2a0f5db4
Aug 08 04:39:08 dist-test-slave-drl0 krb5kdc[7774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163948, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:39:08.955302 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:08.935267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:41795 (local address 127.15.228.126:43567)
0808 04:39:08.935491 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:08.935496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:08.936942 (+ 1446us) server_negotiation.cc:407] Connection header received
0808 04:39:08.939312 (+ 2370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:08.939317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:08.939406 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:08.939539 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:08.941539 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:08.942591 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:08.944592 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:08.944886 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:08.945010 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:08.949291 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:08.949310 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:08.949319 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:08.949393 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:08.951598 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:08.952622 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:08.952629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:08.952634 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:08.952753 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:08.953331 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:08.953337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:08.953340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:08.953782 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:08.953955 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:08.954843 (+ 888us) server_negotiation.cc:299] Negotiation successful
0808 04:39:08.955055 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
WARNING: no policy specified for kudu/127.15.228.66@KRBTEST.COM; defaulting to no policy
I20260808 04:39:08.956604 8100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:08.935882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:43567 (local address 127.15.228.65:41795)
0808 04:39:08.936701 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:08.936785 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:08.938812 (+ 2027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:08.939857 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:08.939868 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:08.940429 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:08.941205 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:08.941227 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:08.942837 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:08.942842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:08.944215 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:08.944235 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:08.945118 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:08.945125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:08.945424 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:08.946341 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:08.946374 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:08.948920 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:08.951774 (+ 2854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:08.951780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:08.951808 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:08.952418 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:08.952920 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:08.952924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:08.952926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:08.953168 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:08.954019 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:08.954027 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:08.954445 (+ 418us) client_negotiation.cc:769] Sending connection context
0808 04:39:08.954713 (+ 268us) client_negotiation.cc:240] Negotiation successful
0808 04:39:08.955267 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":193,"threads_started":1}
I20260808 04:39:08.958720 8096 heartbeater.cc:347] Connected to a master server at 127.15.228.126:43567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.66@KRBTEST.COM" created.
I20260808 04:39:08.959197 8096 heartbeater.cc:464] Registering TS with master...
I20260808 04:39:08.960193 8096 heartbeater.cc:511] Master 127.15.228.126:43567 requested a full tablet report, sending...
I20260808 04:39:08.962903 7840 ts_manager.cc:194] Registered new tserver with Master: 75f42f3d6cff4a8e86f48a9339f810de (127.15.228.65:44387)
I20260808 04:39:08.964449 7840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:41795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
I20260808 04:39:08.980515 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:43567
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:09.122989 8105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:09.123333 8105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:09.123413 8105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:09.127898 8105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:09.128001 8105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:09.128139 8105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:39:09.135551 8105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:43567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:09.136662 8105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:09.138052 8105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:09.152057 8114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:09.152071 8112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:09.152051 8116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:09.152433 8105 server_base.cc:1034] running on GCE node
I20260808 04:39:09.152905 8105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:09.153581 8105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:09.156968 8105 hybrid_clock.cc:648] HybridClock initialized: now 1786163949156938 us; error 63 us; skew 500 ppm
Aug 08 04:39:09 dist-test-slave-drl0 krb5kdc[7774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163949, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:09.161108 8105 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:39:09.162587 8105 webserver.cc:460] Webserver started at http://127.15.228.66:40167/ using document root <none> and password file <none>
I20260808 04:39:09.163542 8105 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:09.163662 8105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:09.163960 8105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:09.166644 8105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "995920d0704b4b5c910c156993992df9"
format_stamp: "Formatted at 2026-08-08 04:39:09 on dist-test-slave-drl0"
server_key: "2ee53a5a353ffdc9eb3d12919e8fbe06"
server_key_iv: "8b80566db53f5223b102a98f2ecd15e0"
server_key_version: "encryptionkey@0"
I20260808 04:39:09.167677 8105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "995920d0704b4b5c910c156993992df9"
format_stamp: "Formatted at 2026-08-08 04:39:09 on dist-test-slave-drl0"
server_key: "2ee53a5a353ffdc9eb3d12919e8fbe06"
server_key_iv: "8b80566db53f5223b102a98f2ecd15e0"
server_key_version: "encryptionkey@0"
I20260808 04:39:09.172252 8105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:39:09.175045 8127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:09.176641 8105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:39:09.176826 8105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "995920d0704b4b5c910c156993992df9"
format_stamp: "Formatted at 2026-08-08 04:39:09 on dist-test-slave-drl0"
server_key: "2ee53a5a353ffdc9eb3d12919e8fbe06"
server_key_iv: "8b80566db53f5223b102a98f2ecd15e0"
server_key_version: "encryptionkey@0"
I20260808 04:39:09.176959 8105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:09.188325 8105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:09.191710 8105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:09.191932 8105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:09.193037 8105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:39:09.194180 8105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:39:09.194262 8105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:09.194341 8105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:39:09.194393 8105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:09.206568 8105 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:39217
I20260808 04:39:09.206593 8275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:39217 every 8 connection(s)
I20260808 04:39:09.208014 8105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:39:09.210872 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 8105
I20260808 04:39:09.211057 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:39:09.211405 16273 external_mini_cluster.cc:1428] Setting key 04cf10701f15d7e3c11738bbb4a5942c
Aug 08 04:39:09 dist-test-slave-drl0 krb5kdc[7774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163949, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.228.67@KRBTEST.COM; defaulting to no policy
I20260808 04:39:09.226770 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:09.210642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:58437 (local address 127.15.228.126:43567)
0808 04:39:09.210804 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:09.210808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:09.212019 (+ 1211us) server_negotiation.cc:407] Connection header received
0808 04:39:09.214124 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:09.214133 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:09.214232 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:09.214404 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:09.215981 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.216714 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.217929 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.218153 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.218285 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:09.221997 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:09.222002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:09.222005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:09.222044 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:09.224154 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:09.224833 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:09.224837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:09.224839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:09.224934 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:09.225318 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:09.225321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:09.225323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:09.225618 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:09.225735 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:09.226434 (+ 699us) server_negotiation.cc:299] Negotiation successful
0808 04:39:09.226582 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.67@KRBTEST.COM" created.
I20260808 04:39:09.227733 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:09.210854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:43567 (local address 127.15.228.66:58437)
0808 04:39:09.211823 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:09.211886 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:09.213811 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:09.214598 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:09.214605 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:09.215109 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:09.215722 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:09.215782 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.216879 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.216883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:09.217784 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:09.217795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.218333 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.218338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:09.218517 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:09.219485 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:09.219502 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:09.221644 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:09.224325 (+ 2681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:09.224333 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:09.224355 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:09.224710 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:09.225043 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:09.225047 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:09.225049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:09.225201 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:09.225734 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:09.225743 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:09.226115 (+ 372us) client_negotiation.cc:769] Sending connection context
0808 04:39:09.226367 (+ 252us) client_negotiation.cc:240] Negotiation successful
0808 04:39:09.226731 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":219,"threads_started":1}
I20260808 04:39:09.229214 8276 heartbeater.cc:347] Connected to a master server at 127.15.228.126:43567
I20260808 04:39:09.229562 8276 heartbeater.cc:464] Registering TS with master...
I20260808 04:39:09.230199 8276 heartbeater.cc:511] Master 127.15.228.126:43567 requested a full tablet report, sending...
I20260808 04:39:09.231575 7840 ts_manager.cc:194] Registered new tserver with Master: 995920d0704b4b5c910c156993992df9 (127.15.228.66:39217)
I20260808 04:39:09.232226 7840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:58437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
I20260808 04:39:09.247107 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:43567
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:09.374822 8284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:09.375242 8284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:09.375406 8284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:09.381597 8284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:09.381672 8284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:09.381743 8284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:39:09.387956 8284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:43567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:09.389034 8284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:09.390071 8284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:09.401868 8291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:09.401866 8294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:09.401922 8284 server_base.cc:1034] running on GCE node
W20260808 04:39:09.401866 8290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:09.402746 8284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:09.403663 8284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:09.404897 8284 hybrid_clock.cc:648] HybridClock initialized: now 1786163949404837 us; error 67 us; skew 500 ppm
Aug 08 04:39:09 dist-test-slave-drl0 krb5kdc[7774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163949, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:09.408962 8284 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:39:09.410238 8284 webserver.cc:460] Webserver started at http://127.15.228.67:40421/ using document root <none> and password file <none>
I20260808 04:39:09.410892 8284 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:09.411042 8284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:09.411247 8284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:09.413421 8284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "7e45b1fc674a49e180d37f7532c6bd51"
format_stamp: "Formatted at 2026-08-08 04:39:09 on dist-test-slave-drl0"
server_key: "73d82bef024006dc9be090aeb34af3a2"
server_key_iv: "a198fad0a9f929dc918be1ec4ac10a6a"
server_key_version: "encryptionkey@0"
I20260808 04:39:09.413964 8284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "7e45b1fc674a49e180d37f7532c6bd51"
format_stamp: "Formatted at 2026-08-08 04:39:09 on dist-test-slave-drl0"
server_key: "73d82bef024006dc9be090aeb34af3a2"
server_key_iv: "a198fad0a9f929dc918be1ec4ac10a6a"
server_key_version: "encryptionkey@0"
I20260808 04:39:09.418197 8284 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260808 04:39:09.420910 8306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:09.422564 8284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:39:09.422677 8284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "7e45b1fc674a49e180d37f7532c6bd51"
format_stamp: "Formatted at 2026-08-08 04:39:09 on dist-test-slave-drl0"
server_key: "73d82bef024006dc9be090aeb34af3a2"
server_key_iv: "a198fad0a9f929dc918be1ec4ac10a6a"
server_key_version: "encryptionkey@0"
I20260808 04:39:09.422837 8284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:09.439474 8284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:09.443581 8284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:09.443813 8284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:09.444597 8284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:39:09.445703 8284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:39:09.445785 8284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:09.445854 8284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:39:09.445885 8284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:09.456343 8284 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:42713
I20260808 04:39:09.456427 8456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:42713 every 8 connection(s)
I20260808 04:39:09.457338 8284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:39:09.466355 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 8284
I20260808 04:39:09.466513 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:39:09.466842 16273 external_mini_cluster.cc:1428] Setting key 59f201c5286a2cf6b1caba849960d988
Aug 08 04:39:09 dist-test-slave-drl0 krb5kdc[7774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163949, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:39:09.475122 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:09.460110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:46697 (local address 127.15.228.126:43567)
0808 04:39:09.460337 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:09.460341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:09.461205 (+ 864us) server_negotiation.cc:407] Connection header received
0808 04:39:09.462648 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:09.462652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:09.462715 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:09.462831 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:09.464557 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.465136 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.466706 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.466991 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.467122 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:09.470206 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:09.470210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:09.470212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:09.470236 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:09.472678 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:09.473265 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:09.473268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:09.473270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:09.473351 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:09.473792 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:09.473795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:09.473796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:09.474022 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:09.474120 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:09.474803 (+ 683us) server_negotiation.cc:299] Negotiation successful
0808 04:39:09.474914 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260808 04:39:09.476616 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:09.460495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:43567 (local address 127.15.228.67:46697)
0808 04:39:09.461035 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:09.461090 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:09.462418 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:09.463044 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:09.463050 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:09.463616 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:09.464346 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:09.464365 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.465316 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.465320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:09.466425 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:09.466559 (+ 134us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.467172 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.467176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:09.467380 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:09.468048 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:09.468065 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:09.470015 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:09.472849 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:09.472854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:09.472868 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:09.473140 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:09.473480 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:09.473483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:09.473485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:09.473661 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:09.474164 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:09.474174 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:09.474526 (+ 352us) client_negotiation.cc:769] Sending connection context
0808 04:39:09.474711 (+ 185us) client_negotiation.cc:240] Negotiation successful
0808 04:39:09.475378 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":128,"threads_started":1}
I20260808 04:39:09.478472 8457 heartbeater.cc:347] Connected to a master server at 127.15.228.126:43567
I20260808 04:39:09.478737 8457 heartbeater.cc:464] Registering TS with master...
I20260808 04:39:09.479326 8457 heartbeater.cc:511] Master 127.15.228.126:43567 requested a full tablet report, sending...
I20260808 04:39:09.481005 7840 ts_manager.cc:194] Registered new tserver with Master: 7e45b1fc674a49e180d37f7532c6bd51 (127.15.228.67:42713)
I20260808 04:39:09.482116 7840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:46697
I20260808 04:39:09.482795 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:39:09.494820 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:09.485664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58994 (local address 127.15.228.126:43567)
0808 04:39:09.485789 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:09.485794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:09.485817 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:39:09.485959 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:09.485962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:09.486002 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:09.486079 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:09.487202 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.487733 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.488728 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.488948 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.489036 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:09.491046 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:09.491050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:09.491052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:09.491075 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:09.492868 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:09.493414 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:09.493418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:09.493420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:09.493517 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:09.493867 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:09.493870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:09.493872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:09.494125 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:09.494257 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:09.494585 (+ 328us) server_negotiation.cc:299] Negotiation successful
0808 04:39:09.494692 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:39:09.498515 7840 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:39:09.503193 7840 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:39:09.521610 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:09.514567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39848 (local address 127.15.228.66:39217)
0808 04:39:09.515090 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:09.515104 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:09.515509 (+ 405us) server_negotiation.cc:407] Connection header received
0808 04:39:09.515739 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:09.515745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:09.516309 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:09.516448 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:09.516710 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.517848 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.519704 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.520418 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.520546 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:09.520715 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:09.521398 (+ 683us) server_negotiation.cc:299] Negotiation successful
0808 04:39:09.521487 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":124,"threads_started":1}
I20260808 04:39:09.521627 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:09.514560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:42713 (local address 127.0.0.1:53416)
0808 04:39:09.515404 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:09.515421 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:09.515555 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:09.516250 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:09.516256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:09.516281 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:09.516448 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:09.516457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.518018 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.518021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:09.519877 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:09.519887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.521136 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.521151 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:09.521228 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:09.521250 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 04:39:09.521346 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 04:39:09.521477 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":630,"thread_start_us":92,"threads_started":1}
I20260808 04:39:09.521636 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:09.515214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53416 (local address 127.15.228.67:42713)
0808 04:39:09.515735 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:09.515748 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:09.515773 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:39:09.515858 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:09.515864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:09.516078 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:09.516267 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:09.516581 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.517854 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.520007 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.520985 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.521130 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:09.521263 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:09.521383 (+ 120us) server_negotiation.cc:299] Negotiation successful
0808 04:39:09.521476 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":182,"threads_started":1}
I20260808 04:39:09.522226 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:09.514317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:39217 (local address 127.0.0.1:39848)
0808 04:39:09.515349 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:09.515419 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:09.515569 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:09.516466 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:09.516469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:09.516488 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:09.516580 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:09.516586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.518018 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.518021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:09.519565 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:09.519577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.520939 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.520957 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:09.521101 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:09.521164 (+ 63us) client_negotiation.cc:769] Sending connection context
0808 04:39:09.521274 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 04:39:09.522071 (+ 797us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":374,"thread_start_us":111,"threads_started":1}
I20260808 04:39:09.522408 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:09.515493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:44387 (local address 127.0.0.1:39202)
0808 04:39:09.515882 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:09.515895 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:09.515981 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:09.516642 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:09.516645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:09.516663 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:09.516775 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:09.516780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.517990 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.517994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:09.519517 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:09.519529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.520904 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.520908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:09.522096 (+ 1188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:09.522130 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 04:39:09.522204 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 04:39:09.522277 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":1142,"thread_start_us":91,"threads_started":1}
I20260808 04:39:09.522897 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:09.515910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39202 (local address 127.15.228.65:44387)
0808 04:39:09.516189 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:09.516195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:09.516217 (+ 22us) server_negotiation.cc:407] Connection header received
0808 04:39:09.516292 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:09.516296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:09.516455 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:09.516584 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:09.516902 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.517847 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.519662 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.520564 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.520677 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:09.520814 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:09.522633 (+ 1819us) server_negotiation.cc:299] Negotiation successful
0808 04:39:09.522744 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":118,"threads_started":1}
I20260808 04:39:09.525043 8367 tablet_service.cc:1467] Processing CreateTablet for tablet 376d882874c44bdaa964f65653252d92 (DEFAULT_TABLE table=test-table [id=edaedbd82c7b45c083c181187826446b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:39:09.525110 8188 tablet_service.cc:1467] Processing CreateTablet for tablet 376d882874c44bdaa964f65653252d92 (DEFAULT_TABLE table=test-table [id=edaedbd82c7b45c083c181187826446b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:39:09.525398 8013 tablet_service.cc:1467] Processing CreateTablet for tablet 376d882874c44bdaa964f65653252d92 (DEFAULT_TABLE table=test-table [id=edaedbd82c7b45c083c181187826446b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:39:09.526105 8367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 376d882874c44bdaa964f65653252d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:09.526191 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 376d882874c44bdaa964f65653252d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:09.526242 8013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 376d882874c44bdaa964f65653252d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:09.534021 8476 tablet_bootstrap.cc:492] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9: Bootstrap starting.
I20260808 04:39:09.535378 8477 tablet_bootstrap.cc:492] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51: Bootstrap starting.
I20260808 04:39:09.535718 8480 tablet_bootstrap.cc:492] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de: Bootstrap starting.
I20260808 04:39:09.538134 8477 tablet_bootstrap.cc:654] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:09.538134 8476 tablet_bootstrap.cc:654] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:09.538146 8480 tablet_bootstrap.cc:654] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:09.539373 8480 log.cc:824] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:09.539780 8476 log.cc:824] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:09.539780 8477 log.cc:824] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:09.542178 8477 tablet_bootstrap.cc:492] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51: No bootstrap required, opened a new log
I20260808 04:39:09.542179 8480 tablet_bootstrap.cc:492] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de: No bootstrap required, opened a new log
I20260808 04:39:09.542418 8480 ts_tablet_manager.cc:1397] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:39:09.542418 8477 ts_tablet_manager.cc:1397] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:39:09.542533 8476 tablet_bootstrap.cc:492] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9: No bootstrap required, opened a new log
I20260808 04:39:09.542869 8476 ts_tablet_manager.cc:1397] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260808 04:39:09.546085 8476 raft_consensus.cc:348] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995920d0704b4b5c910c156993992df9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39217 } } peers { permanent_uuid: "7e45b1fc674a49e180d37f7532c6bd51" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42713 } } peers { permanent_uuid: "75f42f3d6cff4a8e86f48a9339f810de" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 44387 } }
I20260808 04:39:09.546423 8476 raft_consensus.cc:374] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:09.546502 8476 raft_consensus.cc:729] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 995920d0704b4b5c910c156993992df9, State: Initialized, Role: FOLLOWER
I20260808 04:39:09.546653 8477 raft_consensus.cc:348] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995920d0704b4b5c910c156993992df9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39217 } } peers { permanent_uuid: "7e45b1fc674a49e180d37f7532c6bd51" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42713 } } peers { permanent_uuid: "75f42f3d6cff4a8e86f48a9339f810de" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 44387 } }
I20260808 04:39:09.546908 8477 raft_consensus.cc:374] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:09.546983 8477 raft_consensus.cc:729] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e45b1fc674a49e180d37f7532c6bd51, State: Initialized, Role: FOLLOWER
I20260808 04:39:09.547046 8476 consensus_queue.cc:260] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995920d0704b4b5c910c156993992df9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39217 } } peers { permanent_uuid: "7e45b1fc674a49e180d37f7532c6bd51" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42713 } } peers { permanent_uuid: "75f42f3d6cff4a8e86f48a9339f810de" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 44387 } }
I20260808 04:39:09.547130 8480 raft_consensus.cc:348] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995920d0704b4b5c910c156993992df9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39217 } } peers { permanent_uuid: "7e45b1fc674a49e180d37f7532c6bd51" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42713 } } peers { permanent_uuid: "75f42f3d6cff4a8e86f48a9339f810de" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 44387 } }
I20260808 04:39:09.547379 8480 raft_consensus.cc:374] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:09.547441 8480 raft_consensus.cc:729] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f42f3d6cff4a8e86f48a9339f810de, State: Initialized, Role: FOLLOWER
I20260808 04:39:09.547397 8477 consensus_queue.cc:260] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995920d0704b4b5c910c156993992df9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39217 } } peers { permanent_uuid: "7e45b1fc674a49e180d37f7532c6bd51" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42713 } } peers { permanent_uuid: "75f42f3d6cff4a8e86f48a9339f810de" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 44387 } }
I20260808 04:39:09.548102 8276 heartbeater.cc:503] Master 127.15.228.126:43567 was elected leader, sending a full tablet report...
I20260808 04:39:09.548240 8476 ts_tablet_manager.cc:1428] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:39:09.548342 8477 ts_tablet_manager.cc:1428] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260808 04:39:09.548398 8457 heartbeater.cc:503] Master 127.15.228.126:43567 was elected leader, sending a full tablet report...
I20260808 04:39:09.547876 8480 consensus_queue.cc:260] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995920d0704b4b5c910c156993992df9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39217 } } peers { permanent_uuid: "7e45b1fc674a49e180d37f7532c6bd51" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42713 } } peers { permanent_uuid: "75f42f3d6cff4a8e86f48a9339f810de" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 44387 } }
I20260808 04:39:09.549635 8096 heartbeater.cc:503] Master 127.15.228.126:43567 was elected leader, sending a full tablet report...
I20260808 04:39:09.550031 8480 ts_tablet_manager.cc:1428] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260808 04:39:09.587445 8492 raft_consensus.cc:482] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:39:09.587827 8492 raft_consensus.cc:504] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995920d0704b4b5c910c156993992df9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39217 } } peers { permanent_uuid: "7e45b1fc674a49e180d37f7532c6bd51" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42713 } } peers { permanent_uuid: "75f42f3d6cff4a8e86f48a9339f810de" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 44387 } }
I20260808 04:39:09.589447 8492 leader_election.cc:288] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 995920d0704b4b5c910c156993992df9 (127.15.228.66:39217), 7e45b1fc674a49e180d37f7532c6bd51 (127.15.228.67:42713)
I20260808 04:39:09.595039 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:09.589905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:42713 (local address 127.15.228.65:59183)
0808 04:39:09.590396 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:09.590415 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:09.590591 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:09.591075 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:09.591078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:09.591099 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:09.591323 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:09.591330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.592133 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.592136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:09.593014 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:09.593023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.594029 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.594035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:09.594467 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:09.594561 (+ 94us) client_negotiation.cc:769] Sending connection context
0808 04:39:09.594657 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 04:39:09.594740 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":131,"thread_start_us":181,"threads_started":1}
I20260808 04:39:09.595216 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:09.590005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:33227 (local address 127.15.228.66:39217)
0808 04:39:09.590176 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:09.590188 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:09.590624 (+ 436us) server_negotiation.cc:407] Connection header received
0808 04:39:09.591001 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:09.591005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:09.591070 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:09.591161 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:09.591487 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.591990 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.593156 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.593781 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.593931 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:09.593989 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:09.594829 (+ 840us) server_negotiation.cc:299] Negotiation successful
0808 04:39:09.595012 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:39:09.595039 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:09.589965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:39217 (local address 127.15.228.65:33227)
0808 04:39:09.590451 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:09.590466 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:09.590757 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:09.591194 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:09.591197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:09.591213 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:09.591349 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:09.591355 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.592113 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.592119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:09.593025 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:09.593035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.594029 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.594035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:09.594467 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:09.594561 (+ 94us) client_negotiation.cc:769] Sending connection context
0808 04:39:09.594657 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 04:39:09.594740 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":217,"thread_start_us":209,"threads_started":1}
I20260808 04:39:09.596243 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:09.589996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:59183 (local address 127.15.228.67:42713)
0808 04:39:09.590251 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:09.590257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:09.590541 (+ 284us) server_negotiation.cc:407] Connection header received
0808 04:39:09.590794 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:09.590802 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:09.590890 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:09.591027 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:09.591447 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.591994 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.593217 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.593815 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.593931 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:09.593989 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:09.595938 (+ 1949us) server_negotiation.cc:299] Negotiation successful
0808 04:39:09.596061 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260808 04:39:09.596388 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376d882874c44bdaa964f65653252d92" candidate_uuid: "75f42f3d6cff4a8e86f48a9339f810de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "995920d0704b4b5c910c156993992df9" is_pre_election: true
I20260808 04:39:09.596781 8214 raft_consensus.cc:2454] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75f42f3d6cff4a8e86f48a9339f810de in term 0.
I20260808 04:39:09.597043 8397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376d882874c44bdaa964f65653252d92" candidate_uuid: "75f42f3d6cff4a8e86f48a9339f810de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e45b1fc674a49e180d37f7532c6bd51" is_pre_election: true
I20260808 04:39:09.597410 8397 raft_consensus.cc:2454] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75f42f3d6cff4a8e86f48a9339f810de in term 0.
I20260808 04:39:09.597463 7948 leader_election.cc:302] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f42f3d6cff4a8e86f48a9339f810de, 995920d0704b4b5c910c156993992df9; no voters:
I20260808 04:39:09.597846 8492 raft_consensus.cc:2781] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:39:09.598001 8492 raft_consensus.cc:482] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:39:09.598052 8492 raft_consensus.cc:3037] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:09.600135 8492 raft_consensus.cc:504] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995920d0704b4b5c910c156993992df9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39217 } } peers { permanent_uuid: "7e45b1fc674a49e180d37f7532c6bd51" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42713 } } peers { permanent_uuid: "75f42f3d6cff4a8e86f48a9339f810de" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 44387 } }
I20260808 04:39:09.600732 8492 leader_election.cc:288] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de [CANDIDATE]: Term 1 election: Requested vote from peers 995920d0704b4b5c910c156993992df9 (127.15.228.66:39217), 7e45b1fc674a49e180d37f7532c6bd51 (127.15.228.67:42713)
I20260808 04:39:09.601261 8214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376d882874c44bdaa964f65653252d92" candidate_uuid: "75f42f3d6cff4a8e86f48a9339f810de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "995920d0704b4b5c910c156993992df9"
I20260808 04:39:09.601449 8214 raft_consensus.cc:3037] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:09.601549 8397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376d882874c44bdaa964f65653252d92" candidate_uuid: "75f42f3d6cff4a8e86f48a9339f810de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e45b1fc674a49e180d37f7532c6bd51"
I20260808 04:39:09.601720 8397 raft_consensus.cc:3037] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:09.602813 8214 raft_consensus.cc:2454] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75f42f3d6cff4a8e86f48a9339f810de in term 1.
I20260808 04:39:09.603119 8397 raft_consensus.cc:2454] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75f42f3d6cff4a8e86f48a9339f810de in term 1.
I20260808 04:39:09.603222 7948 leader_election.cc:302] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f42f3d6cff4a8e86f48a9339f810de, 995920d0704b4b5c910c156993992df9; no voters:
I20260808 04:39:09.603461 8492 raft_consensus.cc:2781] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:09.603747 8492 raft_consensus.cc:686] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de [term 1 LEADER]: Becoming Leader. State: Replica: 75f42f3d6cff4a8e86f48a9339f810de, State: Running, Role: LEADER
I20260808 04:39:09.604151 8492 consensus_queue.cc:237] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995920d0704b4b5c910c156993992df9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39217 } } peers { permanent_uuid: "7e45b1fc674a49e180d37f7532c6bd51" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42713 } } peers { permanent_uuid: "75f42f3d6cff4a8e86f48a9339f810de" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 44387 } }
I20260808 04:39:09.607558 7838 catalog_manager.cc:5495] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de reported cstate change: term changed from 0 to 1, leader changed from <none> to 75f42f3d6cff4a8e86f48a9339f810de (127.15.228.65). New cstate: current_term: 1 leader_uuid: "75f42f3d6cff4a8e86f48a9339f810de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995920d0704b4b5c910c156993992df9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e45b1fc674a49e180d37f7532c6bd51" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75f42f3d6cff4a8e86f48a9339f810de" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 44387 } health_report { overall_health: HEALTHY } } }
I20260808 04:39:09.626824 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:09.622204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.15.228.65:44387)
0808 04:39:09.622342 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:09.622345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:09.622438 (+ 93us) server_negotiation.cc:407] Connection header received
0808 04:39:09.622588 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:09.622591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:09.622646 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:09.622765 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:39:09.623096 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.623606 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.624707 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.624994 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.625175 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:09.625579 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:39:09.626029 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:39:09.626150 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:09.626347 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 04:39:09.626434 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:39:09.634796 8214 raft_consensus.cc:1261] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9 [term 1 FOLLOWER]: Refusing update from remote peer 75f42f3d6cff4a8e86f48a9339f810de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:39:09.634789 8397 raft_consensus.cc:1261] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51 [term 1 FOLLOWER]: Refusing update from remote peer 75f42f3d6cff4a8e86f48a9339f810de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:39:09.635593 8496 consensus_queue.cc:1035] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de [LEADER]: Connected to new peer: Peer: permanent_uuid: "995920d0704b4b5c910c156993992df9" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:39:09.635882 8492 consensus_queue.cc:1035] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e45b1fc674a49e180d37f7532c6bd51" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 42713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:39:09.644249 8503 mvcc.cc:205] Tried to move back new op lower bound from 7316127537694572544 to 7316127537581518848. Current Snapshot: MvccSnapshot[applied={T|T < 7316127537694572544}]
I20260808 04:39:09.646524 8504 mvcc.cc:205] Tried to move back new op lower bound from 7316127537694572544 to 7316127537581518848. Current Snapshot: MvccSnapshot[applied={T|T < 7316127537694572544}]
I20260808 04:39:09.655048 7839 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:39:09.655629 7839 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:39:09.659109 7839 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fef141b050654dbc82b0f7d0e5e6a832: Sending DeleteTablet for 3 replicas of tablet 376d882874c44bdaa964f65653252d92
I20260808 04:39:09.660262 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet 376d882874c44bdaa964f65653252d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:39:09 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:39848
I20260808 04:39:09.660730 8367 tablet_service.cc:1514] Processing DeleteTablet for tablet 376d882874c44bdaa964f65653252d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:39:09 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:53416
I20260808 04:39:09.660730 8013 tablet_service.cc:1514] Processing DeleteTablet for tablet 376d882874c44bdaa964f65653252d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:39:09 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:39202
I20260808 04:39:09.661424 8518 tablet_replica.cc:330] T 376d882874c44bdaa964f65653252d92 P 75f42f3d6cff4a8e86f48a9339f810de: stopping tablet replica
I20260808 04:39:09.661439 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 7925
I20260808 04:39:09.662366 8514 tablet_replica.cc:330] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9: stopping tablet replica
I20260808 04:39:09.661703 8515 tablet_replica.cc:330] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51: stopping tablet replica
I20260808 04:39:09.663002 8515 raft_consensus.cc:2229] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:39:09.663309 8514 raft_consensus.cc:2229] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:39:09.663682 8514 raft_consensus.cc:2258] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:39:09.664275 8515 raft_consensus.cc:2258] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:39:09.666225 8515 ts_tablet_manager.cc:1905] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:39:09.668264 8514 ts_tablet_manager.cc:1905] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:39:09.668838 8515 ts_tablet_manager.cc:1918] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:39:09.668957 8515 log.cc:1196] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/376d882874c44bdaa964f65653252d92
I20260808 04:39:09.669306 8515 ts_tablet_manager.cc:1939] T 376d882874c44bdaa964f65653252d92 P 7e45b1fc674a49e180d37f7532c6bd51: Deleting consensus metadata
I20260808 04:39:09.670521 7822 catalog_manager.cc:4841] TS 7e45b1fc674a49e180d37f7532c6bd51 (127.15.228.67:42713): tablet 376d882874c44bdaa964f65653252d92 (table test-table [id=edaedbd82c7b45c083c181187826446b]) successfully deleted
I20260808 04:39:09.672533 8514 ts_tablet_manager.cc:1918] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:39:09.672652 8514 log.cc:1196] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/376d882874c44bdaa964f65653252d92
I20260808 04:39:09.673014 8514 ts_tablet_manager.cc:1939] T 376d882874c44bdaa964f65653252d92 P 995920d0704b4b5c910c156993992df9: Deleting consensus metadata
I20260808 04:39:09.674413 7824 catalog_manager.cc:4841] TS 995920d0704b4b5c910c156993992df9 (127.15.228.66:39217): tablet 376d882874c44bdaa964f65653252d92 (table test-table [id=edaedbd82c7b45c083c181187826446b]) successfully deleted
W20260808 04:39:09.675511 7822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.228.65:44387 (error 108)
I20260808 04:39:09.676515 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 8105
I20260808 04:39:09.677027 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:09.676579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:44387 (local address 127.0.0.1:39232)
0808 04:39:09.676828 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:09.676918 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.65:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260808 04:39:09.679551 7822 catalog_manager.cc:4584] TS 75f42f3d6cff4a8e86f48a9339f810de (127.15.228.65:44387): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 376d882874c44bdaa964f65653252d92: Network error: Client connection negotiation failed: client connection to 127.15.228.65:44387: connect: Connection refused (error 111)
I20260808 04:39:09.687955 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 8284
I20260808 04:39:09.712167 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 7794
2026-08-08T04:39:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2795 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T04:39:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:09Z Disabled control of system clock
I20260808 04:39:09.761871 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:42209
--webserver_interface=127.15.228.126
--webserver_port=0
--builtin_ntp_servers=127.15.228.84:41151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:42209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:39:09.897007 8523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:09.897418 8523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:09.897462 8523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:09.903343 8523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:09.903465 8523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:09.903483 8523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:09.903503 8523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:09.903522 8523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:09.910193 8523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:41151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:42209
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:42209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:09.912009 8523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:09.914140 8523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:09.926138 8541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:09.926193 8537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:09.926138 8539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:09.926601 8523 server_base.cc:1034] running on GCE node
I20260808 04:39:09.927457 8523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:09.929793 8523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:09.931015 8523 hybrid_clock.cc:648] HybridClock initialized: now 1786163949930980 us; error 46 us; skew 500 ppm
I20260808 04:39:09.933843 8523 webserver.cc:460] Webserver started at http://127.15.228.126:41815/ using document root <none> and password file <none>
I20260808 04:39:09.934703 8523 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:09.934813 8523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:09.935115 8523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:09.938624 8523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "98f174fd2f3940d285baa1d86d8ba4ab"
format_stamp: "Formatted at 2026-08-08 04:39:09 on dist-test-slave-drl0"
server_key: "3664157763b97e5fee3494ae3ffc4bc2"
server_key_iv: "cacb40671f793a291e9f984343da0e24"
server_key_version: "encryptionkey@0"
I20260808 04:39:09.939489 8523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "98f174fd2f3940d285baa1d86d8ba4ab"
format_stamp: "Formatted at 2026-08-08 04:39:09 on dist-test-slave-drl0"
server_key: "3664157763b97e5fee3494ae3ffc4bc2"
server_key_iv: "cacb40671f793a291e9f984343da0e24"
server_key_version: "encryptionkey@0"
I20260808 04:39:09.945389 8523 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260808 04:39:09.949360 8548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:09.951179 8523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:39:09.951367 8523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "98f174fd2f3940d285baa1d86d8ba4ab"
format_stamp: "Formatted at 2026-08-08 04:39:09 on dist-test-slave-drl0"
server_key: "3664157763b97e5fee3494ae3ffc4bc2"
server_key_iv: "cacb40671f793a291e9f984343da0e24"
server_key_version: "encryptionkey@0"
I20260808 04:39:09.951534 8523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:09.966702 8523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:09.967710 8523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:09.967911 8523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:09.975683 8523 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:42209
I20260808 04:39:09.975817 8621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:42209 every 8 connection(s)
I20260808 04:39:09.976756 8523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:09.980150 8622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:09.981775 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 8523
I20260808 04:39:09.981964 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:09.982319 16273 external_mini_cluster.cc:1428] Setting key 1c4e3f5d49935475c41ebe8415d661e8
I20260808 04:39:09.989315 8622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab: Bootstrap starting.
I20260808 04:39:09.993189 8622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:09.994059 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:09.984128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39442 (local address 127.15.228.126:42209)
0808 04:39:09.984841 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:09.984867 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:09.984917 (+ 50us) server_negotiation.cc:407] Connection header received
0808 04:39:09.985623 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:09.985641 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:09.986075 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:09.986461 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:09.986923 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.988078 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.989352 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:09.989919 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:09.990109 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:09.991475 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:09.991505 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:09.991525 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:09.992123 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:09.992211 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:09.992270 (+ 59us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:09.992404 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:09.992620 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 04:39:09.992986 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":142,"threads_started":1}
I20260808 04:39:09.994305 8622 log.cc:824] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:09.997045 8622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab: No bootstrap required, opened a new log
I20260808 04:39:10.001284 8622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f174fd2f3940d285baa1d86d8ba4ab" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42209 } }
I20260808 04:39:10.001744 8622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:10.001806 8622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f174fd2f3940d285baa1d86d8ba4ab, State: Initialized, Role: FOLLOWER
I20260808 04:39:10.002624 8622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f174fd2f3940d285baa1d86d8ba4ab" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42209 } }
I20260808 04:39:10.002990 8622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:10.003126 8622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:10.003273 8622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:10.004827 8622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f174fd2f3940d285baa1d86d8ba4ab" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42209 } }
I20260808 04:39:10.005255 8622 leader_election.cc:302] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98f174fd2f3940d285baa1d86d8ba4ab; no voters:
I20260808 04:39:10.005676 8622 leader_election.cc:288] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:10.005903 8627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:10.006172 8627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab [term 1 LEADER]: Becoming Leader. State: Replica: 98f174fd2f3940d285baa1d86d8ba4ab, State: Running, Role: LEADER
I20260808 04:39:10.006517 8627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f174fd2f3940d285baa1d86d8ba4ab" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42209 } }
I20260808 04:39:10.007251 8622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:10.008780 8628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98f174fd2f3940d285baa1d86d8ba4ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f174fd2f3940d285baa1d86d8ba4ab" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42209 } } }
I20260808 04:39:10.008852 8630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98f174fd2f3940d285baa1d86d8ba4ab. Latest consensus state: current_term: 1 leader_uuid: "98f174fd2f3940d285baa1d86d8ba4ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f174fd2f3940d285baa1d86d8ba4ab" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42209 } } }
I20260808 04:39:10.008939 8628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:10.008939 8630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:10.009920 8642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:10.013291 8642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:10.019968 8642 catalog_manager.cc:1347] Generated new cluster ID: dbc28781792a4105a40a3f1e3ab61a0e
I20260808 04:39:10.020054 8642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:10.040480 8642 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:10.042081 8642 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:10.049322 8642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab: Generated new TSK 0
I20260808 04:39:10.050213 8642 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:10.067153 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:42209
--builtin_ntp_servers=127.15.228.84:41151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:39:10.199844 8652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:10.200289 8652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:10.200356 8652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:10.209093 8652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:10.209198 8652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:10.209312 8652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:39:10.214596 8652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:41151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:42209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:10.215879 8652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:10.217451 8652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:10.228251 8659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:10.228267 8663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:10.228417 8660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:10.228506 8652 server_base.cc:1034] running on GCE node
I20260808 04:39:10.229205 8652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:10.230018 8652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:10.231242 8652 hybrid_clock.cc:648] HybridClock initialized: now 1786163950231191 us; error 59 us; skew 500 ppm
I20260808 04:39:10.233793 8652 webserver.cc:460] Webserver started at http://127.15.228.65:36791/ using document root <none> and password file <none>
I20260808 04:39:10.234411 8652 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:10.234479 8652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:10.234714 8652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:10.237291 8652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "482c878ae254494c945b72655654daa2"
format_stamp: "Formatted at 2026-08-08 04:39:10 on dist-test-slave-drl0"
server_key: "bb22a6266ef325946e587b9b191e72aa"
server_key_iv: "b8da1ca2c07b240345a6d084f00764e8"
server_key_version: "encryptionkey@0"
I20260808 04:39:10.238022 8652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "482c878ae254494c945b72655654daa2"
format_stamp: "Formatted at 2026-08-08 04:39:10 on dist-test-slave-drl0"
server_key: "bb22a6266ef325946e587b9b191e72aa"
server_key_iv: "b8da1ca2c07b240345a6d084f00764e8"
server_key_version: "encryptionkey@0"
I20260808 04:39:10.242586 8652 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:39:10.246428 8671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:10.248196 8652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:39:10.248327 8652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "482c878ae254494c945b72655654daa2"
format_stamp: "Formatted at 2026-08-08 04:39:10 on dist-test-slave-drl0"
server_key: "bb22a6266ef325946e587b9b191e72aa"
server_key_iv: "b8da1ca2c07b240345a6d084f00764e8"
server_key_version: "encryptionkey@0"
I20260808 04:39:10.248571 8652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:10.283936 8652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:10.284722 8652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:10.284938 8652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:10.285588 8652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:39:10.286963 8652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:39:10.287016 8652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:10.287061 8652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:39:10.287077 8652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:10.298254 8652 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:45225
I20260808 04:39:10.298348 8832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:45225 every 8 connection(s)
I20260808 04:39:10.299330 8652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:39:10.307401 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 8652
I20260808 04:39:10.307685 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:39:10.308277 16273 external_mini_cluster.cc:1428] Setting key 91088c0c44d90fbe447251b133345880
I20260808 04:39:10.311414 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:42209
--builtin_ntp_servers=127.15.228.84:41151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:39:10.312330 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.301908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:60719 (local address 127.15.228.126:42209)
0808 04:39:10.302098 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:10.302103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:10.303452 (+ 1349us) server_negotiation.cc:407] Connection header received
0808 04:39:10.304882 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:10.304885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:10.304937 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:10.305026 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:10.306009 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.306793 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.309121 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.309447 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.309588 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.311654 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:10.311658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:10.311662 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:10.311679 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:10.311715 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:10.311724 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:10.311818 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:10.312059 (+ 241us) server_negotiation.cc:299] Negotiation successful
0808 04:39:10.312153 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 04:39:10.313551 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.302304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:42209 (local address 127.15.228.65:60719)
0808 04:39:10.303301 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:10.303346 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:10.304651 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:10.305246 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:10.305253 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:10.305334 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:10.305827 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:10.305840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.307188 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.307192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:10.308843 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:10.308865 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.309706 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.309712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:10.309964 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.311233 (+ 1269us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:39:10.311235 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:10.311299 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:39:10.311398 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:10.311837 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:10.311848 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:10.311858 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 04:39:10.312049 (+ 191us) client_negotiation.cc:240] Negotiation successful
0808 04:39:10.312468 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":156,"threads_started":1}
I20260808 04:39:10.315250 8833 heartbeater.cc:347] Connected to a master server at 127.15.228.126:42209
I20260808 04:39:10.315562 8833 heartbeater.cc:464] Registering TS with master...
I20260808 04:39:10.316617 8833 heartbeater.cc:511] Master 127.15.228.126:42209 requested a full tablet report, sending...
I20260808 04:39:10.318354 8574 ts_manager.cc:194] Registered new tserver with Master: 482c878ae254494c945b72655654daa2 (127.15.228.65:45225)
I20260808 04:39:10.319628 8574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.228.65:60719
W20260808 04:39:10.440598 8836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:10.441053 8836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:10.441186 8836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:10.446169 8836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:10.446269 8836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:10.446374 8836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:39:10.452332 8836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:41151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:42209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:10.454348 8836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:10.456542 8836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:10.468012 8847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:10.467864 8845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:10.467973 8844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:10.468580 8836 server_base.cc:1034] running on GCE node
I20260808 04:39:10.469063 8836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:10.469792 8836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:10.471051 8836 hybrid_clock.cc:648] HybridClock initialized: now 1786163950470995 us; error 63 us; skew 500 ppm
I20260808 04:39:10.473994 8836 webserver.cc:460] Webserver started at http://127.15.228.66:40933/ using document root <none> and password file <none>
I20260808 04:39:10.474874 8836 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:10.475278 8836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:10.475633 8836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:10.478701 8836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "5cd0e0cf39a04e10947a60fcd18d1ece"
format_stamp: "Formatted at 2026-08-08 04:39:10 on dist-test-slave-drl0"
server_key: "33fb28857cb04f8a5bc44350fe9c6722"
server_key_iv: "8e32791968e522a05f4ebb7f8937e8c5"
server_key_version: "encryptionkey@0"
I20260808 04:39:10.479601 8836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "5cd0e0cf39a04e10947a60fcd18d1ece"
format_stamp: "Formatted at 2026-08-08 04:39:10 on dist-test-slave-drl0"
server_key: "33fb28857cb04f8a5bc44350fe9c6722"
server_key_iv: "8e32791968e522a05f4ebb7f8937e8c5"
server_key_version: "encryptionkey@0"
I20260808 04:39:10.484548 8836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:39:10.487564 8854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:10.489112 8836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:39:10.489274 8836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "5cd0e0cf39a04e10947a60fcd18d1ece"
format_stamp: "Formatted at 2026-08-08 04:39:10 on dist-test-slave-drl0"
server_key: "33fb28857cb04f8a5bc44350fe9c6722"
server_key_iv: "8e32791968e522a05f4ebb7f8937e8c5"
server_key_version: "encryptionkey@0"
I20260808 04:39:10.489421 8836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:10.501163 8836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:10.501878 8836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:10.502094 8836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:10.502724 8836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:39:10.503949 8836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:39:10.504033 8836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:10.504117 8836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:39:10.504160 8836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:10.515082 8836 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:40555
I20260808 04:39:10.515125 9009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:40555 every 8 connection(s)
I20260808 04:39:10.516044 8836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:39:10.521059 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 8836
I20260808 04:39:10.521302 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:39:10.521804 16273 external_mini_cluster.cc:1428] Setting key 19d102af569a65a071ee697ad4b64d08
I20260808 04:39:10.524207 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:42209
--builtin_ntp_servers=127.15.228.84:41151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:39:10.530093 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.518235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:43701 (local address 127.15.228.126:42209)
0808 04:39:10.518437 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:10.518441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:10.520290 (+ 1849us) server_negotiation.cc:407] Connection header received
0808 04:39:10.522467 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:10.522471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:10.522532 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:10.522637 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:10.524270 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.525101 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.527427 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.527645 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.527733 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.529499 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:10.529503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:10.529505 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:10.529521 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:10.529542 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:10.529550 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:10.529617 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:10.529904 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 04:39:10.529966 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 04:39:10.531710 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.518866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:42209 (local address 127.15.228.66:43701)
0808 04:39:10.520073 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:10.520133 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:10.522117 (+ 1984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:10.523389 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:10.523399 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:10.523528 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:10.524096 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:10.524111 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.525356 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.525361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:10.527240 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:10.527260 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.527832 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.527839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:10.528106 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.528850 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:39:10.528855 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:10.528923 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:39:10.529017 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:10.529669 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:10.529683 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:10.529693 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 04:39:10.529855 (+ 162us) client_negotiation.cc:240] Negotiation successful
0808 04:39:10.530322 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":177,"threads_started":1}
I20260808 04:39:10.533334 9011 heartbeater.cc:347] Connected to a master server at 127.15.228.126:42209
I20260808 04:39:10.533638 9011 heartbeater.cc:464] Registering TS with master...
I20260808 04:39:10.534914 9011 heartbeater.cc:511] Master 127.15.228.126:42209 requested a full tablet report, sending...
I20260808 04:39:10.536523 8574 ts_manager.cc:194] Registered new tserver with Master: 5cd0e0cf39a04e10947a60fcd18d1ece (127.15.228.66:40555)
I20260808 04:39:10.537472 8574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.228.66:43701
W20260808 04:39:10.656921 9015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:10.657192 9015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:10.657231 9015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:10.661288 9015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:10.661468 9015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:10.661551 9015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:39:10.668958 9015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:41151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:42209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:10.670171 9015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:10.671682 9015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:10.682178 9020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:10.682194 9025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:10.682178 9023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:10.682623 9015 server_base.cc:1034] running on GCE node
I20260808 04:39:10.683109 9015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:10.683761 9015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:10.684970 9015 hybrid_clock.cc:648] HybridClock initialized: now 1786163950684924 us; error 62 us; skew 500 ppm
I20260808 04:39:10.687069 9015 webserver.cc:460] Webserver started at http://127.15.228.67:37239/ using document root <none> and password file <none>
I20260808 04:39:10.687690 9015 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:10.687888 9015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:10.688612 9015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:10.690842 9015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "a79fb8d6b62b4486b94b5c8b8adb4fbd"
format_stamp: "Formatted at 2026-08-08 04:39:10 on dist-test-slave-drl0"
server_key: "3afa8744f6fc76aa5e72b07a8f03eacb"
server_key_iv: "07a96053d92a15815024d7b6aa6c156b"
server_key_version: "encryptionkey@0"
I20260808 04:39:10.691514 9015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "a79fb8d6b62b4486b94b5c8b8adb4fbd"
format_stamp: "Formatted at 2026-08-08 04:39:10 on dist-test-slave-drl0"
server_key: "3afa8744f6fc76aa5e72b07a8f03eacb"
server_key_iv: "07a96053d92a15815024d7b6aa6c156b"
server_key_version: "encryptionkey@0"
I20260808 04:39:10.695765 9015 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:39:10.698845 9034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:10.700762 9015 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:39:10.700934 9015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "a79fb8d6b62b4486b94b5c8b8adb4fbd"
format_stamp: "Formatted at 2026-08-08 04:39:10 on dist-test-slave-drl0"
server_key: "3afa8744f6fc76aa5e72b07a8f03eacb"
server_key_iv: "07a96053d92a15815024d7b6aa6c156b"
server_key_version: "encryptionkey@0"
I20260808 04:39:10.701086 9015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:10.729990 9015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:10.731155 9015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:10.731418 9015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:10.732285 9015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:39:10.733597 9015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:39:10.733680 9015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:10.733788 9015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:39:10.733831 9015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:10.746100 9015 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:36031
I20260808 04:39:10.746186 9189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:36031 every 8 connection(s)
I20260808 04:39:10.747189 9015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:39:10.755457 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 9015
I20260808 04:39:10.755609 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:39:10.756270 16273 external_mini_cluster.cc:1428] Setting key 10d0ad6edcd65c8074589a50a529c0e1
I20260808 04:39:10.759104 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.749717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:33445 (local address 127.15.228.126:42209)
0808 04:39:10.749896 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:10.749900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:10.751008 (+ 1108us) server_negotiation.cc:407] Connection header received
0808 04:39:10.752642 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:10.752645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:10.752693 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:10.752768 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:10.753871 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.754417 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.756402 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.756665 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.756774 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.758525 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:10.758529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:10.758532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:10.758551 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:10.758571 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:10.758579 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:10.758664 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:10.758863 (+ 199us) server_negotiation.cc:299] Negotiation successful
0808 04:39:10.758958 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 04:39:10.760011 9193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.750097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:42209 (local address 127.15.228.67:33445)
0808 04:39:10.750822 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:10.750872 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:10.752411 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:10.752979 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:10.752985 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:10.753065 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:10.753692 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:10.753707 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.754574 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.754577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:10.756144 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:10.756176 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.756933 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.756937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:10.757203 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.758197 (+ 994us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:39:10.758199 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:10.758264 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:39:10.758362 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:10.758686 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:10.758695 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:10.758701 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 04:39:10.758850 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 04:39:10.759173 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":125,"threads_started":1}
I20260808 04:39:10.761438 9190 heartbeater.cc:347] Connected to a master server at 127.15.228.126:42209
I20260808 04:39:10.761656 9190 heartbeater.cc:464] Registering TS with master...
I20260808 04:39:10.762156 9190 heartbeater.cc:511] Master 127.15.228.126:42209 requested a full tablet report, sending...
I20260808 04:39:10.763475 8574 ts_manager.cc:194] Registered new tserver with Master: a79fb8d6b62b4486b94b5c8b8adb4fbd (127.15.228.67:36031)
I20260808 04:39:10.764070 8574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.228.67:33445
I20260808 04:39:10.772246 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:39:10.779978 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.775205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39450 (local address 127.15.228.126:42209)
0808 04:39:10.775428 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:10.775434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:10.775582 (+ 148us) server_negotiation.cc:407] Connection header received
0808 04:39:10.775873 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:10.775877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:10.775940 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:10.776034 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:10.776358 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.776901 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.777786 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.778010 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.778129 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.779519 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:10.779522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:10.779524 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:10.779536 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:10.779556 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:10.779563 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:10.779653 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:10.779751 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 04:39:10.779810 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260808 04:39:10.784248 8574 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:39:10.787753 8574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:39:10.805104 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.799403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:40555 (local address 127.0.0.1:38760)
0808 04:39:10.800222 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:10.800239 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:10.800369 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:10.800961 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:10.800964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:10.800982 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:10.801088 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:10.801099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.802591 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.802595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:10.803571 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:10.803582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.804429 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.804432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:10.804550 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.804592 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 04:39:10.804716 (+ 124us) client_negotiation.cc:240] Negotiation successful
0808 04:39:10.804852 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":44,"thread_start_us":158,"threads_started":1}
I20260808 04:39:10.805083 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.799767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.15.228.66:40555)
0808 04:39:10.800481 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:10.800496 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:10.800520 (+ 24us) server_negotiation.cc:407] Connection header received
0808 04:39:10.800606 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:10.800612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:10.800793 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:10.800973 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:10.801222 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.802442 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.803715 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.804260 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.804417 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.804629 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:10.804747 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 04:39:10.804852 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":281,"threads_started":1}
I20260808 04:39:10.805455 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.798291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:45225 (local address 127.0.0.1:55634)
0808 04:39:10.798628 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:10.798674 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:10.798803 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:10.799844 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:10.799850 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:10.799878 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:10.800055 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:10.800063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.801914 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.801927 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:10.803411 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:10.803421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.804946 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.804951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:10.805066 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.805115 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 04:39:10.805171 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 04:39:10.805270 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260808 04:39:10.806074 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.798631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55634 (local address 127.15.228.65:45225)
0808 04:39:10.799289 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:10.799297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:10.799325 (+ 28us) server_negotiation.cc:407] Connection header received
0808 04:39:10.799413 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:10.799421 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:10.799644 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:10.800221 (+ 577us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:10.800383 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.801664 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.803543 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.804706 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.805528 (+ 822us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.805728 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:10.805796 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 04:39:10.805910 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":144,"threads_started":1}
I20260808 04:39:10.807296 9205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.799536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:36031 (local address 127.0.0.1:44328)
0808 04:39:10.800558 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:10.800571 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:10.800644 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:10.801134 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:10.801140 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:10.801163 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:10.801243 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:10.801249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.802466 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.802470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:10.804997 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:10.805008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.806818 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.806822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:10.806921 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.806968 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 04:39:10.807044 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 04:39:10.807129 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":17,"thread_start_us":143,"threads_started":1}
I20260808 04:39:10.807829 9208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.800053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.15.228.67:36031)
0808 04:39:10.800640 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:10.800651 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:10.800676 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:39:10.800752 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:10.800756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:10.800921 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:10.801084 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:10.801366 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.802323 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.805194 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.806431 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.806545 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.806737 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:10.807547 (+ 810us) server_negotiation.cc:299] Negotiation successful
0808 04:39:10.807655 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":179,"threads_started":1}
I20260808 04:39:10.808832 8742 tablet_service.cc:1467] Processing CreateTablet for tablet 9981b1051b814cc080a19eb5782e3975 (DEFAULT_TABLE table=test-table [id=8a1b0bbfd04f49fe9e9ca486cd411df1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:39:10.808837 8924 tablet_service.cc:1467] Processing CreateTablet for tablet 9981b1051b814cc080a19eb5782e3975 (DEFAULT_TABLE table=test-table [id=8a1b0bbfd04f49fe9e9ca486cd411df1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:39:10.809918 8742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9981b1051b814cc080a19eb5782e3975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:10.809921 8924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9981b1051b814cc080a19eb5782e3975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:10.810473 9102 tablet_service.cc:1467] Processing CreateTablet for tablet 9981b1051b814cc080a19eb5782e3975 (DEFAULT_TABLE table=test-table [id=8a1b0bbfd04f49fe9e9ca486cd411df1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:39:10.811460 9102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9981b1051b814cc080a19eb5782e3975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:10.816994 9209 tablet_bootstrap.cc:492] T 9981b1051b814cc080a19eb5782e3975 P 482c878ae254494c945b72655654daa2: Bootstrap starting.
I20260808 04:39:10.818420 9211 tablet_bootstrap.cc:492] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece: Bootstrap starting.
I20260808 04:39:10.819646 9212 tablet_bootstrap.cc:492] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd: Bootstrap starting.
I20260808 04:39:10.821285 9209 tablet_bootstrap.cc:654] T 9981b1051b814cc080a19eb5782e3975 P 482c878ae254494c945b72655654daa2: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:10.821352 9211 tablet_bootstrap.cc:654] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:10.822279 9211 log.cc:824] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:10.822549 9212 tablet_bootstrap.cc:654] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:10.822654 9209 log.cc:824] T 9981b1051b814cc080a19eb5782e3975 P 482c878ae254494c945b72655654daa2: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:10.823729 9212 log.cc:824] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:10.825147 9211 tablet_bootstrap.cc:492] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece: No bootstrap required, opened a new log
I20260808 04:39:10.825147 9209 tablet_bootstrap.cc:492] T 9981b1051b814cc080a19eb5782e3975 P 482c878ae254494c945b72655654daa2: No bootstrap required, opened a new log
I20260808 04:39:10.825369 9209 ts_tablet_manager.cc:1397] T 9981b1051b814cc080a19eb5782e3975 P 482c878ae254494c945b72655654daa2: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260808 04:39:10.825369 9211 ts_tablet_manager.cc:1397] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 04:39:10.825977 9212 tablet_bootstrap.cc:492] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd: No bootstrap required, opened a new log
I20260808 04:39:10.826195 9212 ts_tablet_manager.cc:1397] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260808 04:39:10.829645 9211 raft_consensus.cc:348] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482c878ae254494c945b72655654daa2" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45225 } } peers { permanent_uuid: "5cd0e0cf39a04e10947a60fcd18d1ece" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 40555 } } peers { permanent_uuid: "a79fb8d6b62b4486b94b5c8b8adb4fbd" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36031 } }
I20260808 04:39:10.829644 9209 raft_consensus.cc:348] T 9981b1051b814cc080a19eb5782e3975 P 482c878ae254494c945b72655654daa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482c878ae254494c945b72655654daa2" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45225 } } peers { permanent_uuid: "5cd0e0cf39a04e10947a60fcd18d1ece" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 40555 } } peers { permanent_uuid: "a79fb8d6b62b4486b94b5c8b8adb4fbd" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36031 } }
I20260808 04:39:10.829939 9211 raft_consensus.cc:374] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:10.829939 9209 raft_consensus.cc:374] T 9981b1051b814cc080a19eb5782e3975 P 482c878ae254494c945b72655654daa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:10.830003 9211 raft_consensus.cc:729] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cd0e0cf39a04e10947a60fcd18d1ece, State: Initialized, Role: FOLLOWER
I20260808 04:39:10.830003 9209 raft_consensus.cc:729] T 9981b1051b814cc080a19eb5782e3975 P 482c878ae254494c945b72655654daa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 482c878ae254494c945b72655654daa2, State: Initialized, Role: FOLLOWER
I20260808 04:39:10.830112 9212 raft_consensus.cc:348] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482c878ae254494c945b72655654daa2" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45225 } } peers { permanent_uuid: "5cd0e0cf39a04e10947a60fcd18d1ece" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 40555 } } peers { permanent_uuid: "a79fb8d6b62b4486b94b5c8b8adb4fbd" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36031 } }
I20260808 04:39:10.830382 9212 raft_consensus.cc:374] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:10.830446 9212 raft_consensus.cc:729] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a79fb8d6b62b4486b94b5c8b8adb4fbd, State: Initialized, Role: FOLLOWER
I20260808 04:39:10.830482 9211 consensus_queue.cc:260] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482c878ae254494c945b72655654daa2" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45225 } } peers { permanent_uuid: "5cd0e0cf39a04e10947a60fcd18d1ece" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 40555 } } peers { permanent_uuid: "a79fb8d6b62b4486b94b5c8b8adb4fbd" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36031 } }
I20260808 04:39:10.830482 9209 consensus_queue.cc:260] T 9981b1051b814cc080a19eb5782e3975 P 482c878ae254494c945b72655654daa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482c878ae254494c945b72655654daa2" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45225 } } peers { permanent_uuid: "5cd0e0cf39a04e10947a60fcd18d1ece" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 40555 } } peers { permanent_uuid: "a79fb8d6b62b4486b94b5c8b8adb4fbd" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36031 } }
I20260808 04:39:10.830935 9212 consensus_queue.cc:260] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482c878ae254494c945b72655654daa2" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45225 } } peers { permanent_uuid: "5cd0e0cf39a04e10947a60fcd18d1ece" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 40555 } } peers { permanent_uuid: "a79fb8d6b62b4486b94b5c8b8adb4fbd" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36031 } }
I20260808 04:39:10.831493 9011 heartbeater.cc:503] Master 127.15.228.126:42209 was elected leader, sending a full tablet report...
I20260808 04:39:10.831610 9211 ts_tablet_manager.cc:1428] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260808 04:39:10.831610 9209 ts_tablet_manager.cc:1428] T 9981b1051b814cc080a19eb5782e3975 P 482c878ae254494c945b72655654daa2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260808 04:39:10.831876 9212 ts_tablet_manager.cc:1428] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260808 04:39:10.832175 9190 heartbeater.cc:503] Master 127.15.228.126:42209 was elected leader, sending a full tablet report...
I20260808 04:39:10.832595 8833 heartbeater.cc:503] Master 127.15.228.126:42209 was elected leader, sending a full tablet report...
I20260808 04:39:10.875988 9216 raft_consensus.cc:482] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:39:10.876348 9216 raft_consensus.cc:504] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482c878ae254494c945b72655654daa2" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45225 } } peers { permanent_uuid: "5cd0e0cf39a04e10947a60fcd18d1ece" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 40555 } } peers { permanent_uuid: "a79fb8d6b62b4486b94b5c8b8adb4fbd" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36031 } }
I20260808 04:39:10.878136 9216 leader_election.cc:288] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 482c878ae254494c945b72655654daa2 (127.15.228.65:45225), a79fb8d6b62b4486b94b5c8b8adb4fbd (127.15.228.67:36031)
I20260808 04:39:10.880326 9219 raft_consensus.cc:482] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:39:10.880573 9219 raft_consensus.cc:504] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482c878ae254494c945b72655654daa2" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45225 } } peers { permanent_uuid: "5cd0e0cf39a04e10947a60fcd18d1ece" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 40555 } } peers { permanent_uuid: "a79fb8d6b62b4486b94b5c8b8adb4fbd" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36031 } }
I20260808 04:39:10.881809 9219 leader_election.cc:288] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 482c878ae254494c945b72655654daa2 (127.15.228.65:45225), 5cd0e0cf39a04e10947a60fcd18d1ece (127.15.228.66:40555)
I20260808 04:39:10.883044 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.878515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:45225 (local address 127.15.228.66:47463)
0808 04:39:10.879073 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:10.879090 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:10.879213 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:10.879471 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:10.879477 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:10.879497 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:10.879601 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:10.879605 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.880589 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.880595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:10.881540 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:10.881551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.882546 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.882551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:10.882662 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.882694 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 04:39:10.882742 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 04:39:10.882831 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":23,"spinlock_wait_cycles":1280,"thread_start_us":164,"threads_started":1}
I20260808 04:39:10.883674 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.878426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:36031 (local address 127.15.228.66:50401)
0808 04:39:10.878721 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:10.878738 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:10.878857 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:10.879186 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:10.879189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:10.879215 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:10.879362 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:10.879368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.880443 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.880447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:10.881629 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:10.881640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.883298 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.883302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:10.883411 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.883434 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:39:10.883483 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 04:39:10.883548 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":44,"spinlock_wait_cycles":3712}
I20260808 04:39:10.883991 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.878813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:47463 (local address 127.15.228.65:45225)
0808 04:39:10.878970 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:10.878975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:10.879209 (+ 234us) server_negotiation.cc:407] Connection header received
0808 04:39:10.879317 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:10.879320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:10.879364 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:10.879425 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:10.879709 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.880429 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.881681 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.882382 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.883529 (+ 1147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.883577 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:10.883611 (+ 34us) server_negotiation.cc:299] Negotiation successful
0808 04:39:10.883675 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:39:10.884552 9208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.878776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:50401 (local address 127.15.228.67:36031)
0808 04:39:10.878918 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:10.878946 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:10.878962 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:39:10.879021 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:10.879026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:10.879075 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:10.879143 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:10.879503 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.880311 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.882296 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.883163 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.883966 (+ 803us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.884294 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:10.884356 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 04:39:10.884434 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":43}
I20260808 04:39:10.884642 8767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9981b1051b814cc080a19eb5782e3975" candidate_uuid: "5cd0e0cf39a04e10947a60fcd18d1ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "482c878ae254494c945b72655654daa2" is_pre_election: true
I20260808 04:39:10.884955 8767 raft_consensus.cc:2454] T 9981b1051b814cc080a19eb5782e3975 P 482c878ae254494c945b72655654daa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cd0e0cf39a04e10947a60fcd18d1ece in term 0.
I20260808 04:39:10.885006 9128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9981b1051b814cc080a19eb5782e3975" candidate_uuid: "5cd0e0cf39a04e10947a60fcd18d1ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a79fb8d6b62b4486b94b5c8b8adb4fbd" is_pre_election: true
I20260808 04:39:10.885178 9128 raft_consensus.cc:2454] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cd0e0cf39a04e10947a60fcd18d1ece in term 0.
I20260808 04:39:10.885555 8857 leader_election.cc:302] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 482c878ae254494c945b72655654daa2, 5cd0e0cf39a04e10947a60fcd18d1ece, a79fb8d6b62b4486b94b5c8b8adb4fbd; no voters:
I20260808 04:39:10.885845 9216 raft_consensus.cc:2781] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:39:10.885944 9216 raft_consensus.cc:482] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:39:10.885990 9216 raft_consensus.cc:3037] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:10.886571 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.882395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:43353 (local address 127.15.228.66:40555)
0808 04:39:10.882502 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:10.882507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:10.882522 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:39:10.882653 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:10.882656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:10.882697 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:10.882762 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:10.883153 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.883801 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.885373 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.885883 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.885992 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.886050 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:10.886413 (+ 363us) server_negotiation.cc:299] Negotiation successful
0808 04:39:10.886478 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 04:39:10.886587 9193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.881936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:45225 (local address 127.15.228.67:33133)
0808 04:39:10.882092 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:10.882105 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:10.882174 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:10.882741 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:10.882744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:10.882762 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:10.882867 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:10.882872 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.883974 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.883977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:10.885203 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:10.885215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.886227 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.886232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:10.886334 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.886365 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 04:39:10.886429 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 04:39:10.886474 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":91}
I20260808 04:39:10.886597 9222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.882038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:40555 (local address 127.15.228.67:43353)
0808 04:39:10.882446 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:10.882462 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:10.882544 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:10.882850 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:10.882853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:10.882870 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:10.883015 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:10.883022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.883923 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.883926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:10.885208 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:10.885223 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.886127 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.886131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:10.886267 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.886296 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 04:39:10.886376 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 04:39:10.886459 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":42,"thread_start_us":132,"threads_started":1}
I20260808 04:39:10.887176 9223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.882085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:33133 (local address 127.15.228.65:45225)
0808 04:39:10.882503 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:10.882512 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:10.882525 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:39:10.882568 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:10.882571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:10.882613 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:10.883001 (+ 388us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:10.883128 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.883851 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.885497 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.886103 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.886241 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.886285 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:10.886973 (+ 688us) server_negotiation.cc:299] Negotiation successful
0808 04:39:10.887044 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":94,"threads_started":1}
I20260808 04:39:10.887230 8954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9981b1051b814cc080a19eb5782e3975" candidate_uuid: "a79fb8d6b62b4486b94b5c8b8adb4fbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cd0e0cf39a04e10947a60fcd18d1ece" is_pre_election: true
I20260808 04:39:10.887480 9216 raft_consensus.cc:504] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482c878ae254494c945b72655654daa2" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45225 } } peers { permanent_uuid: "5cd0e0cf39a04e10947a60fcd18d1ece" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 40555 } } peers { permanent_uuid: "a79fb8d6b62b4486b94b5c8b8adb4fbd" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36031 } }
I20260808 04:39:10.887647 8767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9981b1051b814cc080a19eb5782e3975" candidate_uuid: "a79fb8d6b62b4486b94b5c8b8adb4fbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "482c878ae254494c945b72655654daa2" is_pre_election: true
I20260808 04:39:10.887781 8954 raft_consensus.cc:2379] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a79fb8d6b62b4486b94b5c8b8adb4fbd in current term 1: Already voted for candidate 5cd0e0cf39a04e10947a60fcd18d1ece in this term.
I20260808 04:39:10.887794 8767 raft_consensus.cc:2454] T 9981b1051b814cc080a19eb5782e3975 P 482c878ae254494c945b72655654daa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a79fb8d6b62b4486b94b5c8b8adb4fbd in term 0.
I20260808 04:39:10.887907 9216 leader_election.cc:288] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [CANDIDATE]: Term 1 election: Requested vote from peers 482c878ae254494c945b72655654daa2 (127.15.228.65:45225), a79fb8d6b62b4486b94b5c8b8adb4fbd (127.15.228.67:36031)
I20260808 04:39:10.888289 9128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9981b1051b814cc080a19eb5782e3975" candidate_uuid: "5cd0e0cf39a04e10947a60fcd18d1ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a79fb8d6b62b4486b94b5c8b8adb4fbd"
I20260808 04:39:10.888316 8767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9981b1051b814cc080a19eb5782e3975" candidate_uuid: "5cd0e0cf39a04e10947a60fcd18d1ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "482c878ae254494c945b72655654daa2"
I20260808 04:39:10.888370 9039 leader_election.cc:302] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 482c878ae254494c945b72655654daa2, a79fb8d6b62b4486b94b5c8b8adb4fbd; no voters:
I20260808 04:39:10.888437 8767 raft_consensus.cc:3037] T 9981b1051b814cc080a19eb5782e3975 P 482c878ae254494c945b72655654daa2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:10.888442 9128 raft_consensus.cc:3037] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:10.889531 8767 raft_consensus.cc:2454] T 9981b1051b814cc080a19eb5782e3975 P 482c878ae254494c945b72655654daa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cd0e0cf39a04e10947a60fcd18d1ece in term 1.
I20260808 04:39:10.889549 9128 raft_consensus.cc:2454] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cd0e0cf39a04e10947a60fcd18d1ece in term 1.
I20260808 04:39:10.889645 9219 raft_consensus.cc:2741] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260808 04:39:10.889932 8855 leader_election.cc:302] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cd0e0cf39a04e10947a60fcd18d1ece, a79fb8d6b62b4486b94b5c8b8adb4fbd; no voters:
I20260808 04:39:10.890566 9216 raft_consensus.cc:2781] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:10.890765 9216 raft_consensus.cc:686] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [term 1 LEADER]: Becoming Leader. State: Replica: 5cd0e0cf39a04e10947a60fcd18d1ece, State: Running, Role: LEADER
I20260808 04:39:10.891129 9216 consensus_queue.cc:237] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482c878ae254494c945b72655654daa2" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45225 } } peers { permanent_uuid: "5cd0e0cf39a04e10947a60fcd18d1ece" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 40555 } } peers { permanent_uuid: "a79fb8d6b62b4486b94b5c8b8adb4fbd" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36031 } }
I20260808 04:39:10.894615 8574 catalog_manager.cc:5495] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cd0e0cf39a04e10947a60fcd18d1ece (127.15.228.66). New cstate: current_term: 1 leader_uuid: "5cd0e0cf39a04e10947a60fcd18d1ece" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "482c878ae254494c945b72655654daa2" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cd0e0cf39a04e10947a60fcd18d1ece" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 40555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a79fb8d6b62b4486b94b5c8b8adb4fbd" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36031 } health_report { overall_health: UNKNOWN } } }
I20260808 04:39:10.908295 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.904356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38768 (local address 127.15.228.66:40555)
0808 04:39:10.904507 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:10.904512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:10.904528 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:39:10.904706 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:10.904716 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:10.904780 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:10.904893 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:39:10.905132 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.905746 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.906611 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:10.906917 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:10.907059 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:10.907380 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:39:10.907842 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:39:10.908013 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:10.908057 (+ 44us) server_negotiation.cc:299] Negotiation successful
0808 04:39:10.908132 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:39:10.914860 8767 raft_consensus.cc:1261] T 9981b1051b814cc080a19eb5782e3975 P 482c878ae254494c945b72655654daa2 [term 1 FOLLOWER]: Refusing update from remote peer 5cd0e0cf39a04e10947a60fcd18d1ece: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:39:10.915179 9128 raft_consensus.cc:1261] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd [term 1 FOLLOWER]: Refusing update from remote peer 5cd0e0cf39a04e10947a60fcd18d1ece: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:39:10.915720 9216 consensus_queue.cc:1035] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [LEADER]: Connected to new peer: Peer: permanent_uuid: "482c878ae254494c945b72655654daa2" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 45225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:39:10.916116 9224 consensus_queue.cc:1035] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [LEADER]: Connected to new peer: Peer: permanent_uuid: "a79fb8d6b62b4486b94b5c8b8adb4fbd" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 36031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:39:10.926651 9228 mvcc.cc:205] Tried to move back new op lower bound from 7316127542938116096 to 7316127542852608000. Current Snapshot: MvccSnapshot[applied={T|T < 7316127542938116096}]
I20260808 04:39:10.926873 9227 mvcc.cc:205] Tried to move back new op lower bound from 7316127542938116096 to 7316127542852608000. Current Snapshot: MvccSnapshot[applied={T|T < 7316127542938116096}]
I20260808 04:39:10.940538 8574 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:39:10.940750 8574 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:39:10.943466 8574 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98f174fd2f3940d285baa1d86d8ba4ab: Sending DeleteTablet for 3 replicas of tablet 9981b1051b814cc080a19eb5782e3975
I20260808 04:39:10.944397 8924 tablet_service.cc:1514] Processing DeleteTablet for tablet 9981b1051b814cc080a19eb5782e3975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:39:10 UTC) from {username='slave'} at 127.0.0.1:38760
I20260808 04:39:10.944463 8742 tablet_service.cc:1514] Processing DeleteTablet for tablet 9981b1051b814cc080a19eb5782e3975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:39:10 UTC) from {username='slave'} at 127.0.0.1:55634
I20260808 04:39:10.944943 9242 tablet_replica.cc:330] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece: stopping tablet replica
I20260808 04:39:10.945237 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 8652
I20260808 04:39:10.945210 9102 tablet_service.cc:1514] Processing DeleteTablet for tablet 9981b1051b814cc080a19eb5782e3975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:39:10 UTC) from {username='slave'} at 127.0.0.1:44328
I20260808 04:39:10.945300 9242 raft_consensus.cc:2229] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:39:10.945757 9242 raft_consensus.cc:2258] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:39:10.949826 9245 tablet_replica.cc:330] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd: stopping tablet replica
I20260808 04:39:10.950078 9242 ts_tablet_manager.cc:1905] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:39:10.951973 9245 raft_consensus.cc:2229] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:39:10.952416 9245 raft_consensus.cc:2258] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:39:10.954869 9242 ts_tablet_manager.cc:1918] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:39:10.955070 9242 log.cc:1196] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/9981b1051b814cc080a19eb5782e3975
I20260808 04:39:10.955443 9242 ts_tablet_manager.cc:1939] T 9981b1051b814cc080a19eb5782e3975 P 5cd0e0cf39a04e10947a60fcd18d1ece: Deleting consensus metadata
I20260808 04:39:10.955614 9245 ts_tablet_manager.cc:1905] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:39:10.956907 8551 catalog_manager.cc:4841] TS 5cd0e0cf39a04e10947a60fcd18d1ece (127.15.228.66:40555): tablet 9981b1051b814cc080a19eb5782e3975 (table test-table [id=8a1b0bbfd04f49fe9e9ca486cd411df1]) successfully deleted
I20260808 04:39:10.958750 9245 ts_tablet_manager.cc:1918] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:39:10.958913 9245 log.cc:1196] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/9981b1051b814cc080a19eb5782e3975
I20260808 04:39:10.959347 9245 ts_tablet_manager.cc:1939] T 9981b1051b814cc080a19eb5782e3975 P a79fb8d6b62b4486b94b5c8b8adb4fbd: Deleting consensus metadata
I20260808 04:39:10.960652 8550 catalog_manager.cc:4841] TS a79fb8d6b62b4486b94b5c8b8adb4fbd (127.15.228.67:36031): tablet 9981b1051b814cc080a19eb5782e3975 (table test-table [id=8a1b0bbfd04f49fe9e9ca486cd411df1]) successfully deleted
W20260808 04:39:10.961254 8552 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.228.65:45225 (error 108)
I20260808 04:39:10.961486 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 8836
I20260808 04:39:10.980674 9205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:10.980058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:45225 (local address 127.0.0.1:55642)
0808 04:39:10.980405 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:10.980515 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.65:45225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
W20260808 04:39:10.981177 8552 catalog_manager.cc:4584] TS 482c878ae254494c945b72655654daa2 (127.15.228.65:45225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9981b1051b814cc080a19eb5782e3975: Network error: Client connection negotiation failed: client connection to 127.15.228.65:45225: connect: Connection refused (error 111)
I20260808 04:39:10.993078 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 9015
I20260808 04:39:11.008479 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 8523
2026-08-08T04:39:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1285 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:11 dist-test-slave-drl0 krb5kdc[9250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:11 dist-test-slave-drl0 krb5kdc[9250](info): set up 2 sockets
Aug 08 04:39:11 dist-test-slave-drl0 krb5kdc[9250](info): commencing operation
krb5kdc: starting...
W20260808 04:39:12.305541 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.270s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:12 dist-test-slave-drl0 krb5kdc[9250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:39:12.471628 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:44623
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:44707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:44623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:12.603729 9270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:12.604050 9270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:12.604096 9270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:12.608682 9270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:12.608759 9270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:12.608785 9270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:12.608803 9270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:12.608819 9270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:12.614637 9270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:44707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:44623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:44623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:12.615927 9270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:12.617059 9270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:12.628638 9275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:12.628577 9279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:12.628607 9277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:12.629361 9270 server_base.cc:1034] running on GCE node
I20260808 04:39:12.630091 9270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:12.631497 9270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:12.633052 9270 hybrid_clock.cc:648] HybridClock initialized: now 1786163952633010 us; error 74 us; skew 500 ppm
Aug 08 04:39:12 dist-test-slave-drl0 krb5kdc[9250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163952, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:12.640635 9270 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:39:12.641780 9270 webserver.cc:460] Webserver started at http://127.15.228.126:34157/ using document root <none> and password file <none>
I20260808 04:39:12.642421 9270 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:12.642517 9270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:12.642779 9270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:12.645006 9270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "e5e7ce568ec2419e82ad1721de9723c3"
format_stamp: "Formatted at 2026-08-08 04:39:12 on dist-test-slave-drl0"
server_key: "f0b9dfdfa0c232ff637dbba75be3f575"
server_key_iv: "e8d09ceb1a871c28985c752ba1bae4b0"
server_key_version: "encryptionkey@0"
I20260808 04:39:12.645606 9270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "e5e7ce568ec2419e82ad1721de9723c3"
format_stamp: "Formatted at 2026-08-08 04:39:12 on dist-test-slave-drl0"
server_key: "f0b9dfdfa0c232ff637dbba75be3f575"
server_key_iv: "e8d09ceb1a871c28985c752ba1bae4b0"
server_key_version: "encryptionkey@0"
I20260808 04:39:12.649595 9270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:39:12.652809 9289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:12.654567 9270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:39:12.654750 9270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "e5e7ce568ec2419e82ad1721de9723c3"
format_stamp: "Formatted at 2026-08-08 04:39:12 on dist-test-slave-drl0"
server_key: "f0b9dfdfa0c232ff637dbba75be3f575"
server_key_iv: "e8d09ceb1a871c28985c752ba1bae4b0"
server_key_version: "encryptionkey@0"
I20260808 04:39:12.654888 9270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:12.683496 9270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:12.687973 9270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:12.688429 9270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:12.698249 9270 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:44623
I20260808 04:39:12.698284 9360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:44623 every 8 connection(s)
I20260808 04:39:12.699354 9270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:12.702123 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 9270
I20260808 04:39:12.702250 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:12.702569 9361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:12.702621 16273 external_mini_cluster.cc:1428] Setting key da93f5f58ae818d54957918d71c9df5f
I20260808 04:39:12.709625 9361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3: Bootstrap starting.
I20260808 04:39:12.713418 9361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:12.714457 9361 log.cc:824] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:39:12 dist-test-slave-drl0 krb5kdc[9250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:39:12.718488 9361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3: No bootstrap required, opened a new log
I20260808 04:39:12.724609 9361 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e7ce568ec2419e82ad1721de9723c3" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 44623 } }
I20260808 04:39:12.724994 9361 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:12.725068 9361 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e7ce568ec2419e82ad1721de9723c3, State: Initialized, Role: FOLLOWER
I20260808 04:39:12.725663 9361 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e7ce568ec2419e82ad1721de9723c3" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 44623 } }
I20260808 04:39:12.725811 9361 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:12.725905 9361 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:12.726030 9361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:12.726845 9364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:12.705424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35984 (local address 127.15.228.126:44623)
0808 04:39:12.705914 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:12.705935 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:12.705982 (+ 47us) server_negotiation.cc:407] Connection header received
0808 04:39:12.706846 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:12.706869 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:12.707308 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:12.707693 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:12.708992 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:12.710426 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:12.711888 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:12.713047 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:12.713247 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:12.718011 (+ 4764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:12.718017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:12.718033 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:12.718063 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:12.720422 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:12.722381 (+ 1959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:12.722388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:12.722399 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:12.723372 (+ 973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:12.723814 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:12.723817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:12.723818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:12.724581 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:12.724777 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:12.725525 (+ 748us) server_negotiation.cc:299] Negotiation successful
0808 04:39:12.725875 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":130,"threads_started":1}
I20260808 04:39:12.727149 9361 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e7ce568ec2419e82ad1721de9723c3" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 44623 } }
I20260808 04:39:12.728324 9361 leader_election.cc:302] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5e7ce568ec2419e82ad1721de9723c3; no voters:
I20260808 04:39:12.728962 9361 leader_election.cc:288] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:12.729082 9366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:12.729650 9366 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3 [term 1 LEADER]: Becoming Leader. State: Replica: e5e7ce568ec2419e82ad1721de9723c3, State: Running, Role: LEADER
I20260808 04:39:12.729964 9366 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e7ce568ec2419e82ad1721de9723c3" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 44623 } }
I20260808 04:39:12.730490 9361 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:12.731751 9368 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5e7ce568ec2419e82ad1721de9723c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e7ce568ec2419e82ad1721de9723c3" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 44623 } } }
I20260808 04:39:12.731889 9368 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:12.732256 9369 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5e7ce568ec2419e82ad1721de9723c3. Latest consensus state: current_term: 1 leader_uuid: "e5e7ce568ec2419e82ad1721de9723c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e7ce568ec2419e82ad1721de9723c3" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 44623 } } }
I20260808 04:39:12.732369 9369 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:12.732462 9377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:12.736173 9377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:12.744585 9377 catalog_manager.cc:1347] Generated new cluster ID: 70dcdd31d1e942ea8768850ad39e5dde
I20260808 04:39:12.744683 9377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:12.760540 9377 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:12.762108 9377 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:12.775560 9377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3: Generated new TSK 0
I20260808 04:39:12.776652 9377 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
I20260808 04:39:12.828967 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:44623
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:44707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:12.984059 9400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:12.984450 9400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:12.984539 9400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:12.988976 9400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:12.989099 9400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:12.989202 9400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:39:12.997648 9400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:44707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:44623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:12.999418 9400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:13.001331 9400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:13.013437 9408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:13.013518 9400 server_base.cc:1034] running on GCE node
W20260808 04:39:13.013468 9410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:13.013463 9407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:13.014370 9400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:13.015192 9400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:13.016443 9400 hybrid_clock.cc:648] HybridClock initialized: now 1786163953016428 us; error 31 us; skew 500 ppm
Aug 08 04:39:13 dist-test-slave-drl0 krb5kdc[9250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163953, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:13.020881 9400 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:39:13.022250 9400 webserver.cc:460] Webserver started at http://127.15.228.65:33389/ using document root <none> and password file <none>
I20260808 04:39:13.022895 9400 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:13.023049 9400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:13.023260 9400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:13.026585 9400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "30bdaf8a7a7d48e1bd910cfc6777dcfe"
format_stamp: "Formatted at 2026-08-08 04:39:13 on dist-test-slave-drl0"
server_key: "9dd5e262c165fe11bf591a316df22a81"
server_key_iv: "59717db6b4c36f6e041145dfa69b6f83"
server_key_version: "encryptionkey@0"
I20260808 04:39:13.027453 9400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "30bdaf8a7a7d48e1bd910cfc6777dcfe"
format_stamp: "Formatted at 2026-08-08 04:39:13 on dist-test-slave-drl0"
server_key: "9dd5e262c165fe11bf591a316df22a81"
server_key_iv: "59717db6b4c36f6e041145dfa69b6f83"
server_key_version: "encryptionkey@0"
I20260808 04:39:13.032829 9400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:39:13.036262 9419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:13.038175 9400 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:39:13.038429 9400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "30bdaf8a7a7d48e1bd910cfc6777dcfe"
format_stamp: "Formatted at 2026-08-08 04:39:13 on dist-test-slave-drl0"
server_key: "9dd5e262c165fe11bf591a316df22a81"
server_key_iv: "59717db6b4c36f6e041145dfa69b6f83"
server_key_version: "encryptionkey@0"
I20260808 04:39:13.038563 9400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:13.090322 9400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:13.094225 9400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:13.094480 9400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:13.095201 9400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:39:13.096128 9400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:39:13.096204 9400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:13.096277 9400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:39:13.096329 9400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:13.110287 9400 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:36403
I20260808 04:39:13.110510 9585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:36403 every 8 connection(s)
I20260808 04:39:13.111968 9400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:39:13.113219 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 9400
I20260808 04:39:13.113390 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:39:13.113732 16273 external_mini_cluster.cc:1428] Setting key b7ffc848eb4fd43b9573301b47d800ab
Aug 08 04:39:13 dist-test-slave-drl0 krb5kdc[9250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163953, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.228.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.66@KRBTEST.COM" created.
I20260808 04:39:13.132082 9364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:13.115341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:47823 (local address 127.15.228.126:44623)
0808 04:39:13.115509 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:13.115516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:13.117133 (+ 1617us) server_negotiation.cc:407] Connection header received
0808 04:39:13.118584 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:13.118587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:13.118645 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:13.118736 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:13.120115 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.120682 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.122017 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.122258 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.122385 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:13.125507 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:13.125511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:13.125516 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:13.125556 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:13.127915 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:13.128962 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:13.128967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:13.128969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:13.129191 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:13.129832 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:13.129838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:13.129840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:13.130416 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:13.130756 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:13.131694 (+ 938us) server_negotiation.cc:299] Negotiation successful
0808 04:39:13.131934 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 04:39:13.133250 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:13.115883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:44623 (local address 127.15.228.65:47823)
0808 04:39:13.116934 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:13.117003 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:13.118330 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:13.119002 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:13.119013 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:13.119436 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:13.119959 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:13.119973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.120824 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.120828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:13.121877 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:13.121887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.122429 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.122433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:13.122609 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:13.123320 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:13.123337 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:13.125291 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:13.128157 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:13.128166 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:13.128188 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:13.128792 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:13.129409 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:13.129415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:13.129417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:13.129706 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:13.130771 (+ 1065us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:13.130783 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:13.131260 (+ 477us) client_negotiation.cc:769] Sending connection context
0808 04:39:13.131541 (+ 281us) client_negotiation.cc:240] Negotiation successful
0808 04:39:13.132010 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":566,"threads_started":1}
I20260808 04:39:13.134856 9587 heartbeater.cc:347] Connected to a master server at 127.15.228.126:44623
I20260808 04:39:13.135152 9587 heartbeater.cc:464] Registering TS with master...
I20260808 04:39:13.135739 9587 heartbeater.cc:511] Master 127.15.228.126:44623 requested a full tablet report, sending...
I20260808 04:39:13.137403 9308 ts_manager.cc:194] Registered new tserver with Master: 30bdaf8a7a7d48e1bd910cfc6777dcfe (127.15.228.65:36403)
I20260808 04:39:13.138594 9308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:47823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
I20260808 04:39:13.153007 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:44623
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:44707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:13.285158 9594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:13.285449 9594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:13.285494 9594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:13.289638 9594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:13.289721 9594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:13.289783 9594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:39:13.295823 9594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:44707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:44623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:13.297263 9594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:13.298470 9594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:13.310078 9606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:13.310056 9611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:13.310041 9603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:13.310509 9594 server_base.cc:1034] running on GCE node
I20260808 04:39:13.311041 9594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:13.311748 9594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:13.312978 9594 hybrid_clock.cc:648] HybridClock initialized: now 1786163953312957 us; error 36 us; skew 500 ppm
Aug 08 04:39:13 dist-test-slave-drl0 krb5kdc[9250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163953, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:13.316666 9594 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:39:13.318173 9594 webserver.cc:460] Webserver started at http://127.15.228.66:34429/ using document root <none> and password file <none>
I20260808 04:39:13.319067 9594 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:13.319196 9594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:13.319505 9594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:13.322156 9594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "6803792403154a5ea6a376d6193e68f3"
format_stamp: "Formatted at 2026-08-08 04:39:13 on dist-test-slave-drl0"
server_key: "4a23e90b822eaeec58a80b7019ec666f"
server_key_iv: "7167edaa2dfd27dedcf4beb2bc205f21"
server_key_version: "encryptionkey@0"
I20260808 04:39:13.322818 9594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "6803792403154a5ea6a376d6193e68f3"
format_stamp: "Formatted at 2026-08-08 04:39:13 on dist-test-slave-drl0"
server_key: "4a23e90b822eaeec58a80b7019ec666f"
server_key_iv: "7167edaa2dfd27dedcf4beb2bc205f21"
server_key_version: "encryptionkey@0"
I20260808 04:39:13.327811 9594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 04:39:13.331702 9618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:13.333252 9594 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:39:13.333374 9594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "6803792403154a5ea6a376d6193e68f3"
format_stamp: "Formatted at 2026-08-08 04:39:13 on dist-test-slave-drl0"
server_key: "4a23e90b822eaeec58a80b7019ec666f"
server_key_iv: "7167edaa2dfd27dedcf4beb2bc205f21"
server_key_version: "encryptionkey@0"
I20260808 04:39:13.333475 9594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:13.353497 9594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:13.357369 9594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:13.357548 9594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:13.358157 9594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:39:13.359170 9594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:39:13.359253 9594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:13.359328 9594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:39:13.359382 9594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:13.372481 9594 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:36441
I20260808 04:39:13.372614 9771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:36441 every 8 connection(s)
I20260808 04:39:13.373482 9594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:39:13.382860 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 9594
I20260808 04:39:13.383082 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:39:13.383409 16273 external_mini_cluster.cc:1428] Setting key 6009c321a80484c67282215a33c64c45
Aug 08 04:39:13 dist-test-slave-drl0 krb5kdc[9250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163953, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:39:13.391624 9364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:13.375779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:56553 (local address 127.15.228.126:44623)
0808 04:39:13.375976 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:13.375981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:13.376920 (+ 939us) server_negotiation.cc:407] Connection header received
0808 04:39:13.378770 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:13.378774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:13.378846 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:13.378976 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:13.380189 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.380742 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.382180 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.382488 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.382623 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:13.386403 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:13.386411 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:13.386417 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:13.386459 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:13.388491 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:13.389337 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:13.389342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:13.389343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:13.389534 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:13.390088 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:13.390091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:13.390093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:13.390313 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:13.390431 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:13.391258 (+ 827us) server_negotiation.cc:299] Negotiation successful
0808 04:39:13.391475 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260808 04:39:13.392803 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:13.376120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:44623 (local address 127.15.228.66:56553)
0808 04:39:13.376646 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:13.376694 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:13.378517 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:13.379166 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:13.379173 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:13.379545 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:13.380040 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:13.380053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.380890 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.380899 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:13.382015 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:13.382027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.382676 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.382681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:13.382871 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:13.383727 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:13.383746 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:13.386136 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:13.388640 (+ 2504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:13.388650 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:13.388672 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:13.389106 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:13.389703 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:13.389716 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:13.389722 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:13.389937 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:13.390461 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:13.390471 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:13.390854 (+ 383us) client_negotiation.cc:769] Sending connection context
0808 04:39:13.391162 (+ 308us) client_negotiation.cc:240] Negotiation successful
0808 04:39:13.391581 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":123,"threads_started":1}
I20260808 04:39:13.394304 9772 heartbeater.cc:347] Connected to a master server at 127.15.228.126:44623
I20260808 04:39:13.394605 9772 heartbeater.cc:464] Registering TS with master...
I20260808 04:39:13.395196 9772 heartbeater.cc:511] Master 127.15.228.126:44623 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.228.67@KRBTEST.COM; defaulting to no policy
I20260808 04:39:13.396410 9308 ts_manager.cc:194] Registered new tserver with Master: 6803792403154a5ea6a376d6193e68f3 (127.15.228.66:36441)
I20260808 04:39:13.397014 9308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:56553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
I20260808 04:39:13.420338 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:44623
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:44707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:13.558020 9780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:13.558449 9780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:13.558590 9780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:13.564177 9780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:13.564370 9780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:13.564493 9780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:39:13.571853 9780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:44707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:44623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:13.573091 9780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:13.574177 9780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:13.585364 9789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:13.585395 9786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:13.585604 9787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:13.586007 9780 server_base.cc:1034] running on GCE node
I20260808 04:39:13.586530 9780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:13.587270 9780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:13.588486 9780 hybrid_clock.cc:648] HybridClock initialized: now 1786163953588424 us; error 60 us; skew 500 ppm
Aug 08 04:39:13 dist-test-slave-drl0 krb5kdc[9250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163953, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:13.592509 9780 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:39:13.593643 9780 webserver.cc:460] Webserver started at http://127.15.228.67:43519/ using document root <none> and password file <none>
I20260808 04:39:13.594256 9780 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:13.594363 9780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:13.594619 9780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:13.596884 9780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "255eec6cb12d43c8971ebb9f358948c6"
format_stamp: "Formatted at 2026-08-08 04:39:13 on dist-test-slave-drl0"
server_key: "97dd557ef851f2cdc1341ba0541317ca"
server_key_iv: "40ba30e90dca5fe1dd5d77c490b133b2"
server_key_version: "encryptionkey@0"
I20260808 04:39:13.597478 9780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "255eec6cb12d43c8971ebb9f358948c6"
format_stamp: "Formatted at 2026-08-08 04:39:13 on dist-test-slave-drl0"
server_key: "97dd557ef851f2cdc1341ba0541317ca"
server_key_iv: "40ba30e90dca5fe1dd5d77c490b133b2"
server_key_version: "encryptionkey@0"
I20260808 04:39:13.601609 9780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:39:13.605027 9798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:13.607369 9780 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:39:13.607683 9780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "255eec6cb12d43c8971ebb9f358948c6"
format_stamp: "Formatted at 2026-08-08 04:39:13 on dist-test-slave-drl0"
server_key: "97dd557ef851f2cdc1341ba0541317ca"
server_key_iv: "40ba30e90dca5fe1dd5d77c490b133b2"
server_key_version: "encryptionkey@0"
I20260808 04:39:13.607889 9780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:13.648670 9780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:13.653035 9780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:13.653293 9780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:13.653977 9780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:39:13.654995 9780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:39:13.655064 9780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:13.655136 9780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:39:13.655179 9780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:13.668669 9780 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:43295
I20260808 04:39:13.668690 9961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:43295 every 8 connection(s)
I20260808 04:39:13.669967 9780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:39:13.671315 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 9780
I20260808 04:39:13.671489 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:39:13.671909 16273 external_mini_cluster.cc:1428] Setting key bdf77f54d27bd8e7eb1e318a7e393de0
Aug 08 04:39:13 dist-test-slave-drl0 krb5kdc[9250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163953, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:39:13.691316 9364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:13.672341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:59225 (local address 127.15.228.126:44623)
0808 04:39:13.672506 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:13.672511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:13.674821 (+ 2310us) server_negotiation.cc:407] Connection header received
0808 04:39:13.677099 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:13.677101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:13.677158 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:13.677247 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:13.678772 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.679783 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.681192 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.681583 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.681735 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:13.684924 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:13.684928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:13.684930 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:13.684961 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:13.687323 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:13.688505 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:13.688508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:13.688510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:13.688645 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:13.689307 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:13.689315 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:13.689317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:13.690170 (+ 853us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:13.690296 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:13.691037 (+ 741us) server_negotiation.cc:299] Negotiation successful
0808 04:39:13.691172 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 04:39:13.692401 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:13.673253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:44623 (local address 127.15.228.67:59225)
0808 04:39:13.674463 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:13.674614 (+ 151us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:13.676646 (+ 2032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:13.677545 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:13.677555 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:13.677970 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:13.678510 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:13.678525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.679962 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.679965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:13.680957 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:13.680968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.682009 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.682017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:13.682182 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:13.682903 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:13.682920 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:13.684674 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:13.687595 (+ 2921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:13.687606 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:13.687634 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:13.688321 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:13.688879 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:13.688883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:13.688885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:13.689144 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:13.690352 (+ 1208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:13.690359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:13.690689 (+ 330us) client_negotiation.cc:769] Sending connection context
0808 04:39:13.690882 (+ 193us) client_negotiation.cc:240] Negotiation successful
0808 04:39:13.691360 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":390,"threads_started":1}
I20260808 04:39:13.693971 9963 heartbeater.cc:347] Connected to a master server at 127.15.228.126:44623
I20260808 04:39:13.694350 9963 heartbeater.cc:464] Registering TS with master...
I20260808 04:39:13.694916 9963 heartbeater.cc:511] Master 127.15.228.126:44623 requested a full tablet report, sending...
I20260808 04:39:13.696281 9308 ts_manager.cc:194] Registered new tserver with Master: 255eec6cb12d43c8971ebb9f358948c6 (127.15.228.67:43295)
I20260808 04:39:13.697024 9308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:59225
I20260808 04:39:13.702675 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:39:13.716758 9364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:13.706055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40320 (local address 127.15.228.126:44623)
0808 04:39:13.706378 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:13.706385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:13.706616 (+ 231us) server_negotiation.cc:407] Connection header received
0808 04:39:13.706715 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:13.706719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:13.706790 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:13.707015 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:13.707971 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.708954 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.710344 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.710972 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.711092 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:13.713133 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:13.713137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:13.713140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:13.713169 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:13.714736 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:13.715292 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:13.715295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:13.715296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:13.715378 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:13.715775 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:13.715778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:13.715779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:13.716008 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:13.716146 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:13.716536 (+ 390us) server_negotiation.cc:299] Negotiation successful
0808 04:39:13.716630 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 04:39:13.720359 9308 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:39:13.723879 9308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:39:13.742179 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:13.735495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:43295 (local address 127.0.0.1:58028)
0808 04:39:13.735884 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:13.735935 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:13.736067 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:13.737147 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:13.737153 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:13.737178 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:13.737376 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:13.737385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.739110 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.739114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:13.740578 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:13.740589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.741616 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.741620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:13.741728 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:13.741766 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 04:39:13.741879 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 04:39:13.742008 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":47,"thread_start_us":128,"threads_started":1}
I20260808 04:39:13.742182 9977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:13.735635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.15.228.67:43295)
0808 04:39:13.736347 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:13.736363 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:13.736405 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:39:13.736639 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:13.736647 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:13.736928 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:13.737543 (+ 615us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:13.737714 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.738971 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.740720 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.741487 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.741615 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:13.741765 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:13.741912 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 04:39:13.742008 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":435,"threads_started":1}
I20260808 04:39:13.742221 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:13.736278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:36403 (local address 127.0.0.1:57118)
0808 04:39:13.737120 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:13.737135 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:13.737294 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:13.737966 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:13.737969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:13.737989 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:13.738081 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:13.738087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.739379 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.739384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:13.740586 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:13.740595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.741772 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.741776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:13.741862 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:13.741890 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 04:39:13.741965 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 04:39:13.742041 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":73,"thread_start_us":131,"threads_started":1}
I20260808 04:39:13.742880 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:13.736461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57118 (local address 127.15.228.65:36403)
0808 04:39:13.736941 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:13.736949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:13.737221 (+ 272us) server_negotiation.cc:407] Connection header received
0808 04:39:13.737455 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:13.737460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:13.737638 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:13.737752 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:13.738214 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.739260 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.740706 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.741444 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.741553 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:13.741707 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:13.742392 (+ 685us) server_negotiation.cc:299] Negotiation successful
0808 04:39:13.742502 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":123,"threads_started":1}
I20260808 04:39:13.744722 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:13.737222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51566 (local address 127.15.228.66:36441)
0808 04:39:13.737758 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:13.737781 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:13.737918 (+ 137us) server_negotiation.cc:407] Connection header received
0808 04:39:13.738021 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:13.738027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:13.738230 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:13.738396 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:13.738680 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.739794 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.741216 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.742056 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.742154 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:13.744255 (+ 2101us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:13.744309 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 04:39:13.744593 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":157,"threads_started":1}
I20260808 04:39:13.745189 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:13.736671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:36441 (local address 127.0.0.1:51566)
0808 04:39:13.737800 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:13.737815 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:13.737915 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:13.738369 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:13.738372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:13.738391 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:13.738491 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:13.738498 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.739952 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.739955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:13.741082 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:13.741092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.743100 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.743106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:13.743306 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:13.743351 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 04:39:13.743466 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 04:39:13.745008 (+ 1542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"thread_start_us":113,"threads_started":1}
I20260808 04:39:13.745719 9868 tablet_service.cc:1467] Processing CreateTablet for tablet 78b9d02b4fe2453d883c27c60f57c9ef (DEFAULT_TABLE table=test-table [id=2a5486975160440eb3af37a8c17633b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:39:13.746791 9868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b9d02b4fe2453d883c27c60f57c9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:13.747984 9682 tablet_service.cc:1467] Processing CreateTablet for tablet 78b9d02b4fe2453d883c27c60f57c9ef (DEFAULT_TABLE table=test-table [id=2a5486975160440eb3af37a8c17633b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:39:13.749073 9682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b9d02b4fe2453d883c27c60f57c9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:13.754005 9491 tablet_service.cc:1467] Processing CreateTablet for tablet 78b9d02b4fe2453d883c27c60f57c9ef (DEFAULT_TABLE table=test-table [id=2a5486975160440eb3af37a8c17633b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:39:13.755198 9491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b9d02b4fe2453d883c27c60f57c9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:13.755653 9983 tablet_bootstrap.cc:492] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3: Bootstrap starting.
I20260808 04:39:13.758477 9983 tablet_bootstrap.cc:654] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:13.760277 9983 log.cc:824] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:13.762542 9987 tablet_bootstrap.cc:492] T 78b9d02b4fe2453d883c27c60f57c9ef P 30bdaf8a7a7d48e1bd910cfc6777dcfe: Bootstrap starting.
I20260808 04:39:13.762773 9983 tablet_bootstrap.cc:492] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3: No bootstrap required, opened a new log
I20260808 04:39:13.763023 9983 ts_tablet_manager.cc:1397] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 04:39:13.763345 9984 tablet_bootstrap.cc:492] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6: Bootstrap starting.
I20260808 04:39:13.767050 9984 tablet_bootstrap.cc:654] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:13.767268 9987 tablet_bootstrap.cc:654] T 78b9d02b4fe2453d883c27c60f57c9ef P 30bdaf8a7a7d48e1bd910cfc6777dcfe: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:13.768538 9987 log.cc:824] T 78b9d02b4fe2453d883c27c60f57c9ef P 30bdaf8a7a7d48e1bd910cfc6777dcfe: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:13.768993 9984 log.cc:824] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:13.771509 9984 tablet_bootstrap.cc:492] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6: No bootstrap required, opened a new log
I20260808 04:39:13.771584 9983 raft_consensus.cc:348] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255eec6cb12d43c8971ebb9f358948c6" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 43295 } } peers { permanent_uuid: "30bdaf8a7a7d48e1bd910cfc6777dcfe" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36403 } } peers { permanent_uuid: "6803792403154a5ea6a376d6193e68f3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 36441 } }
I20260808 04:39:13.771926 9983 raft_consensus.cc:374] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:13.771973 9984 ts_tablet_manager.cc:1397] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260808 04:39:13.771984 9987 tablet_bootstrap.cc:492] T 78b9d02b4fe2453d883c27c60f57c9ef P 30bdaf8a7a7d48e1bd910cfc6777dcfe: No bootstrap required, opened a new log
I20260808 04:39:13.771996 9983 raft_consensus.cc:729] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6803792403154a5ea6a376d6193e68f3, State: Initialized, Role: FOLLOWER
I20260808 04:39:13.772167 9987 ts_tablet_manager.cc:1397] T 78b9d02b4fe2453d883c27c60f57c9ef P 30bdaf8a7a7d48e1bd910cfc6777dcfe: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260808 04:39:13.772578 9983 consensus_queue.cc:260] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255eec6cb12d43c8971ebb9f358948c6" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 43295 } } peers { permanent_uuid: "30bdaf8a7a7d48e1bd910cfc6777dcfe" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36403 } } peers { permanent_uuid: "6803792403154a5ea6a376d6193e68f3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 36441 } }
I20260808 04:39:13.773679 9772 heartbeater.cc:503] Master 127.15.228.126:44623 was elected leader, sending a full tablet report...
I20260808 04:39:13.775223 9983 ts_tablet_manager.cc:1428] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260808 04:39:13.777549 9984 raft_consensus.cc:348] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255eec6cb12d43c8971ebb9f358948c6" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 43295 } } peers { permanent_uuid: "30bdaf8a7a7d48e1bd910cfc6777dcfe" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36403 } } peers { permanent_uuid: "6803792403154a5ea6a376d6193e68f3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 36441 } }
I20260808 04:39:13.777866 9984 raft_consensus.cc:374] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:13.777933 9984 raft_consensus.cc:729] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 255eec6cb12d43c8971ebb9f358948c6, State: Initialized, Role: FOLLOWER
I20260808 04:39:13.778672 9984 consensus_queue.cc:260] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255eec6cb12d43c8971ebb9f358948c6" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 43295 } } peers { permanent_uuid: "30bdaf8a7a7d48e1bd910cfc6777dcfe" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36403 } } peers { permanent_uuid: "6803792403154a5ea6a376d6193e68f3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 36441 } }
I20260808 04:39:13.779507 9963 heartbeater.cc:503] Master 127.15.228.126:44623 was elected leader, sending a full tablet report...
I20260808 04:39:13.779546 9987 raft_consensus.cc:348] T 78b9d02b4fe2453d883c27c60f57c9ef P 30bdaf8a7a7d48e1bd910cfc6777dcfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255eec6cb12d43c8971ebb9f358948c6" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 43295 } } peers { permanent_uuid: "30bdaf8a7a7d48e1bd910cfc6777dcfe" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36403 } } peers { permanent_uuid: "6803792403154a5ea6a376d6193e68f3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 36441 } }
I20260808 04:39:13.779832 9987 raft_consensus.cc:374] T 78b9d02b4fe2453d883c27c60f57c9ef P 30bdaf8a7a7d48e1bd910cfc6777dcfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:13.779884 9984 ts_tablet_manager.cc:1428] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260808 04:39:13.779897 9987 raft_consensus.cc:729] T 78b9d02b4fe2453d883c27c60f57c9ef P 30bdaf8a7a7d48e1bd910cfc6777dcfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30bdaf8a7a7d48e1bd910cfc6777dcfe, State: Initialized, Role: FOLLOWER
I20260808 04:39:13.780324 9987 consensus_queue.cc:260] T 78b9d02b4fe2453d883c27c60f57c9ef P 30bdaf8a7a7d48e1bd910cfc6777dcfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255eec6cb12d43c8971ebb9f358948c6" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 43295 } } peers { permanent_uuid: "30bdaf8a7a7d48e1bd910cfc6777dcfe" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36403 } } peers { permanent_uuid: "6803792403154a5ea6a376d6193e68f3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 36441 } }
I20260808 04:39:13.781261 9587 heartbeater.cc:503] Master 127.15.228.126:44623 was elected leader, sending a full tablet report...
I20260808 04:39:13.781824 9987 ts_tablet_manager.cc:1428] T 78b9d02b4fe2453d883c27c60f57c9ef P 30bdaf8a7a7d48e1bd910cfc6777dcfe: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260808 04:39:13.787038 9991 raft_consensus.cc:482] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:39:13.787786 9991 raft_consensus.cc:504] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255eec6cb12d43c8971ebb9f358948c6" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 43295 } } peers { permanent_uuid: "30bdaf8a7a7d48e1bd910cfc6777dcfe" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36403 } } peers { permanent_uuid: "6803792403154a5ea6a376d6193e68f3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 36441 } }
I20260808 04:39:13.789450 9991 leader_election.cc:288] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 255eec6cb12d43c8971ebb9f358948c6 (127.15.228.67:43295), 30bdaf8a7a7d48e1bd910cfc6777dcfe (127.15.228.65:36403)
I20260808 04:39:13.794628 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:13.789800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:43295 (local address 127.15.228.66:45429)
0808 04:39:13.789968 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:13.789987 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:13.790113 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:13.790519 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:13.790526 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:13.790558 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:13.790898 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:13.790914 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.791958 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.791961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:13.793306 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:13.793320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.794198 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.794201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:13.794289 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:13.794312 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 04:39:13.794386 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 04:39:13.794453 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 04:39:13.794642 9977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:13.790022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:45429 (local address 127.15.228.67:43295)
0808 04:39:13.790186 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:13.790190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:13.790207 (+ 17us) server_negotiation.cc:407] Connection header received
0808 04:39:13.790268 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:13.790270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:13.790323 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:13.790482 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:13.791110 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.791787 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.793485 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.794028 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.794158 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:13.794214 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:13.794427 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 04:39:13.794505 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 04:39:13.795181 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:13.790931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:54989 (local address 127.15.228.65:36403)
0808 04:39:13.791148 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:13.791154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:13.791170 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:39:13.791219 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:13.791222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:13.791269 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:13.791387 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:13.791748 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.792306 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.793870 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.794475 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.794598 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:13.794642 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:13.794921 (+ 279us) server_negotiation.cc:299] Negotiation successful
0808 04:39:13.795032 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260808 04:39:13.795557 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:13.790188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:36403 (local address 127.15.228.66:54989)
0808 04:39:13.790631 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:13.790648 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:13.790756 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:13.791446 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:13.791449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:13.791468 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:13.791611 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:13.791620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.792461 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.792465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:13.793712 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:13.793719 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.794628 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.794632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:13.794747 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:13.794772 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 04:39:13.795339 (+ 567us) client_negotiation.cc:240] Negotiation successful
0808 04:39:13.795417 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":183,"threads_started":1}
I20260808 04:39:13.795487 9893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b9d02b4fe2453d883c27c60f57c9ef" candidate_uuid: "6803792403154a5ea6a376d6193e68f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "255eec6cb12d43c8971ebb9f358948c6" is_pre_election: true
I20260808 04:39:13.795861 9893 raft_consensus.cc:2454] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6803792403154a5ea6a376d6193e68f3 in term 0.
I20260808 04:39:13.796350 9517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b9d02b4fe2453d883c27c60f57c9ef" candidate_uuid: "6803792403154a5ea6a376d6193e68f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30bdaf8a7a7d48e1bd910cfc6777dcfe" is_pre_election: true
I20260808 04:39:13.796676 9621 leader_election.cc:302] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 255eec6cb12d43c8971ebb9f358948c6, 6803792403154a5ea6a376d6193e68f3; no voters:
I20260808 04:39:13.796746 9517 raft_consensus.cc:2454] T 78b9d02b4fe2453d883c27c60f57c9ef P 30bdaf8a7a7d48e1bd910cfc6777dcfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6803792403154a5ea6a376d6193e68f3 in term 0.
I20260808 04:39:13.797119 9991 raft_consensus.cc:2781] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:39:13.797348 9991 raft_consensus.cc:482] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:39:13.797452 9991 raft_consensus.cc:3037] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:13.798748 9991 raft_consensus.cc:504] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255eec6cb12d43c8971ebb9f358948c6" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 43295 } } peers { permanent_uuid: "30bdaf8a7a7d48e1bd910cfc6777dcfe" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36403 } } peers { permanent_uuid: "6803792403154a5ea6a376d6193e68f3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 36441 } }
I20260808 04:39:13.799244 9991 leader_election.cc:288] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [CANDIDATE]: Term 1 election: Requested vote from peers 255eec6cb12d43c8971ebb9f358948c6 (127.15.228.67:43295), 30bdaf8a7a7d48e1bd910cfc6777dcfe (127.15.228.65:36403)
I20260808 04:39:13.799703 9517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b9d02b4fe2453d883c27c60f57c9ef" candidate_uuid: "6803792403154a5ea6a376d6193e68f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30bdaf8a7a7d48e1bd910cfc6777dcfe"
I20260808 04:39:13.799712 9893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b9d02b4fe2453d883c27c60f57c9ef" candidate_uuid: "6803792403154a5ea6a376d6193e68f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "255eec6cb12d43c8971ebb9f358948c6"
I20260808 04:39:13.799814 9517 raft_consensus.cc:3037] T 78b9d02b4fe2453d883c27c60f57c9ef P 30bdaf8a7a7d48e1bd910cfc6777dcfe [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:13.799863 9893 raft_consensus.cc:3037] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:13.800976 9517 raft_consensus.cc:2454] T 78b9d02b4fe2453d883c27c60f57c9ef P 30bdaf8a7a7d48e1bd910cfc6777dcfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6803792403154a5ea6a376d6193e68f3 in term 1.
I20260808 04:39:13.801429 9621 leader_election.cc:302] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30bdaf8a7a7d48e1bd910cfc6777dcfe, 6803792403154a5ea6a376d6193e68f3; no voters:
I20260808 04:39:13.801518 9893 raft_consensus.cc:2454] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6803792403154a5ea6a376d6193e68f3 in term 1.
I20260808 04:39:13.801724 9991 raft_consensus.cc:2781] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:13.802031 9991 raft_consensus.cc:686] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [term 1 LEADER]: Becoming Leader. State: Replica: 6803792403154a5ea6a376d6193e68f3, State: Running, Role: LEADER
I20260808 04:39:13.802835 9991 consensus_queue.cc:237] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255eec6cb12d43c8971ebb9f358948c6" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 43295 } } peers { permanent_uuid: "30bdaf8a7a7d48e1bd910cfc6777dcfe" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36403 } } peers { permanent_uuid: "6803792403154a5ea6a376d6193e68f3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 36441 } }
I20260808 04:39:13.808686 9301 catalog_manager.cc:5495] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6803792403154a5ea6a376d6193e68f3 (127.15.228.66). New cstate: current_term: 1 leader_uuid: "6803792403154a5ea6a376d6193e68f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255eec6cb12d43c8971ebb9f358948c6" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 43295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30bdaf8a7a7d48e1bd910cfc6777dcfe" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6803792403154a5ea6a376d6193e68f3" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 36441 } health_report { overall_health: HEALTHY } } }
I20260808 04:39:13.833487 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:13.828040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51580 (local address 127.15.228.66:36441)
0808 04:39:13.828325 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:13.828330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:13.828348 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:39:13.828429 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:13.828432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:13.828496 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:13.828610 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:39:13.829005 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.829733 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.831219 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:13.831487 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:13.831599 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:13.832151 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:39:13.832662 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:39:13.832796 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:13.833256 (+ 460us) server_negotiation.cc:299] Negotiation successful
0808 04:39:13.833307 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260808 04:39:13.843909 9893 raft_consensus.cc:1261] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6 [term 1 FOLLOWER]: Refusing update from remote peer 6803792403154a5ea6a376d6193e68f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:39:13.844535 9517 raft_consensus.cc:1261] T 78b9d02b4fe2453d883c27c60f57c9ef P 30bdaf8a7a7d48e1bd910cfc6777dcfe [term 1 FOLLOWER]: Refusing update from remote peer 6803792403154a5ea6a376d6193e68f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:39:13.845652 9991 consensus_queue.cc:1035] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bdaf8a7a7d48e1bd910cfc6777dcfe" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:39:13.846290 9995 consensus_queue.cc:1035] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "255eec6cb12d43c8971ebb9f358948c6" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 43295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:39:13.856783 10002 mvcc.cc:205] Tried to move back new op lower bound from 7316127554929721344 to 7316127554783121408. Current Snapshot: MvccSnapshot[applied={T|T < 7316127554929721344}]
I20260808 04:39:13.858700 10001 mvcc.cc:205] Tried to move back new op lower bound from 7316127554929721344 to 7316127554783121408. Current Snapshot: MvccSnapshot[applied={T|T < 7316127554929721344}]
I20260808 04:39:13.868566 9301 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:39:13.868799 9301 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:39:13.872377 9301 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5e7ce568ec2419e82ad1721de9723c3: Sending DeleteTablet for 3 replicas of tablet 78b9d02b4fe2453d883c27c60f57c9ef
I20260808 04:39:13.874863 9682 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b9d02b4fe2453d883c27c60f57c9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:39:13 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:51566
I20260808 04:39:13.875288 9491 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b9d02b4fe2453d883c27c60f57c9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:39:13 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:57118
I20260808 04:39:13.875154 9868 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b9d02b4fe2453d883c27c60f57c9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:39:13 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:58028
I20260808 04:39:13.876235 10014 tablet_replica.cc:330] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3: stopping tablet replica
I20260808 04:39:13.876654 10016 tablet_replica.cc:330] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6: stopping tablet replica
I20260808 04:39:13.877046 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 9400
I20260808 04:39:13.877092 10016 raft_consensus.cc:2229] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:39:13.877475 10016 raft_consensus.cc:2258] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:39:13.886209 10016 ts_tablet_manager.cc:1905] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:39:13.891525 10016 ts_tablet_manager.cc:1918] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:39:13.891676 10016 log.cc:1196] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/78b9d02b4fe2453d883c27c60f57c9ef
I20260808 04:39:13.892172 10016 ts_tablet_manager.cc:1939] T 78b9d02b4fe2453d883c27c60f57c9ef P 255eec6cb12d43c8971ebb9f358948c6: Deleting consensus metadata
I20260808 04:39:13.894909 9292 catalog_manager.cc:4841] TS 255eec6cb12d43c8971ebb9f358948c6 (127.15.228.67:43295): tablet 78b9d02b4fe2453d883c27c60f57c9ef (table test-table [id=2a5486975160440eb3af37a8c17633b4]) successfully deleted
I20260808 04:39:13.896261 10014 raft_consensus.cc:2229] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:39:13.896860 10014 raft_consensus.cc:2258] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:39:13.899508 10014 ts_tablet_manager.cc:1905] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:39:13.903978 10014 ts_tablet_manager.cc:1918] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:39:13.904271 10014 log.cc:1196] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/78b9d02b4fe2453d883c27c60f57c9ef
I20260808 04:39:13.904718 10014 ts_tablet_manager.cc:1939] T 78b9d02b4fe2453d883c27c60f57c9ef P 6803792403154a5ea6a376d6193e68f3: Deleting consensus metadata
W20260808 04:39:13.905279 9292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.228.65:36403 (error 108)
I20260808 04:39:13.905498 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 9594
I20260808 04:39:13.909175 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:13.908830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:36403 (local address 127.0.0.1:57122)
0808 04:39:13.908990 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:13.909079 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.65:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260808 04:39:13.909787 9292 catalog_manager.cc:4584] TS 30bdaf8a7a7d48e1bd910cfc6777dcfe (127.15.228.65:36403): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78b9d02b4fe2453d883c27c60f57c9ef: Network error: Client connection negotiation failed: client connection to 127.15.228.65:36403: connect: Connection refused (error 111)
I20260808 04:39:13.915655 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 9780
I20260808 04:39:13.926667 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:13.916033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:36441 (local address 127.0.0.1:51588)
0808 04:39:13.926412 (+ 10379us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:13.926522 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.66:36441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10200}
I20260808 04:39:13.936213 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 9270
2026-08-08T04:39:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2932 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T04:39:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:13Z Disabled control of system clock
I20260808 04:39:13.980402 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:41373
--webserver_interface=127.15.228.126
--webserver_port=0
--builtin_ntp_servers=127.15.228.84:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:41373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:39:14.119457 10022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:14.119772 10022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:14.119818 10022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:14.124784 10022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:14.124913 10022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:14.124931 10022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:14.124950 10022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:14.124967 10022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:14.133731 10022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:41373
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:41373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:14.135749 10022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:14.138978 10022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:39:14.152247 10022 server_base.cc:1034] running on GCE node
W20260808 04:39:14.152212 10030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:14.152172 10034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:14.152164 10028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:14.153203 10022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:14.154553 10022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:14.158010 10022 hybrid_clock.cc:648] HybridClock initialized: now 1786163954157962 us; error 56 us; skew 500 ppm
I20260808 04:39:14.161175 10022 webserver.cc:460] Webserver started at http://127.15.228.126:36607/ using document root <none> and password file <none>
I20260808 04:39:14.162096 10022 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:14.162231 10022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:14.162537 10022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:14.165073 10022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "22cf737d47e0491c9ccf2d8ebd0da51e"
format_stamp: "Formatted at 2026-08-08 04:39:14 on dist-test-slave-drl0"
server_key: "e5bf4f6c189ac3a0ebe0088ca3cf0842"
server_key_iv: "15c145269fef0c2c9ae755d7c16999a3"
server_key_version: "encryptionkey@0"
I20260808 04:39:14.165752 10022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "22cf737d47e0491c9ccf2d8ebd0da51e"
format_stamp: "Formatted at 2026-08-08 04:39:14 on dist-test-slave-drl0"
server_key: "e5bf4f6c189ac3a0ebe0088ca3cf0842"
server_key_iv: "15c145269fef0c2c9ae755d7c16999a3"
server_key_version: "encryptionkey@0"
I20260808 04:39:14.171554 10022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 04:39:14.178102 10041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:14.180752 10022 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.007s sys 0.000s
I20260808 04:39:14.181032 10022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "22cf737d47e0491c9ccf2d8ebd0da51e"
format_stamp: "Formatted at 2026-08-08 04:39:14 on dist-test-slave-drl0"
server_key: "e5bf4f6c189ac3a0ebe0088ca3cf0842"
server_key_iv: "15c145269fef0c2c9ae755d7c16999a3"
server_key_version: "encryptionkey@0"
I20260808 04:39:14.181202 10022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:14.195191 10022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:14.195994 10022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:14.196211 10022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:14.206444 10022 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:41373
I20260808 04:39:14.206466 10111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:41373 every 8 connection(s)
I20260808 04:39:14.207773 10022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:14.209869 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 10022
I20260808 04:39:14.210050 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:14.210760 16273 external_mini_cluster.cc:1428] Setting key cf95654632b0e98ac1ca22a689e52268
I20260808 04:39:14.213125 10112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:14.221846 10112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e: Bootstrap starting.
I20260808 04:39:14.224373 10116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:14.212753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40444 (local address 127.15.228.126:41373)
0808 04:39:14.214420 (+ 1667us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:14.214448 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:14.214490 (+ 42us) server_negotiation.cc:407] Connection header received
0808 04:39:14.215410 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:14.215432 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:14.215889 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:14.216262 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:14.216753 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:14.217822 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:14.219379 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:14.219995 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:14.220227 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:14.221813 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:14.221820 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:14.221837 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:14.222390 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:14.222470 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:14.222496 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:14.222634 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:14.222818 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 04:39:14.223183 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":195,"threads_started":1}
I20260808 04:39:14.225009 10112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:14.225932 10112 log.cc:824] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:14.228588 10112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e: No bootstrap required, opened a new log
I20260808 04:39:14.234205 10112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22cf737d47e0491c9ccf2d8ebd0da51e" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41373 } }
I20260808 04:39:14.234757 10112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:14.234867 10112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22cf737d47e0491c9ccf2d8ebd0da51e, State: Initialized, Role: FOLLOWER
I20260808 04:39:14.235849 10112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22cf737d47e0491c9ccf2d8ebd0da51e" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41373 } }
I20260808 04:39:14.236095 10112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:14.236284 10112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:14.236406 10112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:14.237903 10112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22cf737d47e0491c9ccf2d8ebd0da51e" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41373 } }
I20260808 04:39:14.238366 10112 leader_election.cc:302] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22cf737d47e0491c9ccf2d8ebd0da51e; no voters:
I20260808 04:39:14.238782 10112 leader_election.cc:288] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:14.239173 10118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:14.239605 10118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e [term 1 LEADER]: Becoming Leader. State: Replica: 22cf737d47e0491c9ccf2d8ebd0da51e, State: Running, Role: LEADER
I20260808 04:39:14.240027 10118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22cf737d47e0491c9ccf2d8ebd0da51e" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41373 } }
I20260808 04:39:14.241071 10112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:14.243019 10120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22cf737d47e0491c9ccf2d8ebd0da51e. Latest consensus state: current_term: 1 leader_uuid: "22cf737d47e0491c9ccf2d8ebd0da51e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22cf737d47e0491c9ccf2d8ebd0da51e" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41373 } } }
I20260808 04:39:14.243091 10119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22cf737d47e0491c9ccf2d8ebd0da51e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22cf737d47e0491c9ccf2d8ebd0da51e" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41373 } } }
I20260808 04:39:14.243247 10120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:14.243263 10119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:14.244062 10124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:14.248188 10124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:14.260500 10124 catalog_manager.cc:1347] Generated new cluster ID: 651a7438e0814c5e9bde63b1d3b55288
I20260808 04:39:14.260648 10124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:14.285738 10124 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:14.289180 10124 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:14.300644 10124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e: Generated new TSK 0
I20260808 04:39:14.302155 10124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:14.310170 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:41373
--builtin_ntp_servers=127.15.228.84:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:39:14.450158 10140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:14.450479 10140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:14.450548 10140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:14.454754 10140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:14.454855 10140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:14.454998 10140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:39:14.462263 10140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:41373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:14.464169 10140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:14.466861 10140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:14.479743 10149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:14.479749 10150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:14.479743 10153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:14.480350 10140 server_base.cc:1034] running on GCE node
I20260808 04:39:14.481410 10140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:14.482149 10140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:14.483359 10140 hybrid_clock.cc:648] HybridClock initialized: now 1786163954483264 us; error 117 us; skew 500 ppm
I20260808 04:39:14.485616 10140 webserver.cc:460] Webserver started at http://127.15.228.65:44065/ using document root <none> and password file <none>
I20260808 04:39:14.486377 10140 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:14.486486 10140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:14.486712 10140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:14.489039 10140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "c5820b6dfc3743bda5ed96b5218387fd"
format_stamp: "Formatted at 2026-08-08 04:39:14 on dist-test-slave-drl0"
server_key: "59b0ae0fff659bf1ccee8110eb194db0"
server_key_iv: "2460a9e6f044449d2c6e6cea78570b19"
server_key_version: "encryptionkey@0"
I20260808 04:39:14.489696 10140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "c5820b6dfc3743bda5ed96b5218387fd"
format_stamp: "Formatted at 2026-08-08 04:39:14 on dist-test-slave-drl0"
server_key: "59b0ae0fff659bf1ccee8110eb194db0"
server_key_iv: "2460a9e6f044449d2c6e6cea78570b19"
server_key_version: "encryptionkey@0"
I20260808 04:39:14.494196 10140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:39:14.497536 10162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:14.499701 10140 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 04:39:14.499862 10140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "c5820b6dfc3743bda5ed96b5218387fd"
format_stamp: "Formatted at 2026-08-08 04:39:14 on dist-test-slave-drl0"
server_key: "59b0ae0fff659bf1ccee8110eb194db0"
server_key_iv: "2460a9e6f044449d2c6e6cea78570b19"
server_key_version: "encryptionkey@0"
I20260808 04:39:14.500028 10140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:14.526307 10140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:14.527473 10140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:14.527730 10140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:14.528725 10140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:39:14.530624 10140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:39:14.530743 10140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:14.530833 10140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:39:14.530882 10140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:14.544274 10317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:36937 every 8 connection(s)
I20260808 04:39:14.544414 10140 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:36937
I20260808 04:39:14.545948 10140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:39:14.551383 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 10140
I20260808 04:39:14.551558 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:39:14.551934 16273 external_mini_cluster.cc:1428] Setting key 739a8425d54fb1dbe6c4ab3ac133679a
I20260808 04:39:14.555099 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:41373
--builtin_ntp_servers=127.15.228.84:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:39:14.558168 10116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:14.548622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:47985 (local address 127.15.228.126:41373)
0808 04:39:14.548824 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:14.548829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:14.550142 (+ 1313us) server_negotiation.cc:407] Connection header received
0808 04:39:14.551943 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:14.551945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:14.552040 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:14.552137 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:14.553301 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:14.554049 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:14.555860 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:14.556100 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:14.556219 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:14.557603 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:14.557614 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:14.557616 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:14.557629 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:14.557647 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:14.557656 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:14.557741 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:14.557937 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 04:39:14.558009 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 04:39:14.559180 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:14.549188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:41373 (local address 127.15.228.65:47985)
0808 04:39:14.549965 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:14.550006 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:14.551670 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:14.552348 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:14.552359 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:14.552469 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:14.553098 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:14.553114 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:14.554261 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:14.554265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:14.555668 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:14.555684 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:14.556281 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:14.556285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:14.556463 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:14.557303 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:39:14.557305 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:14.557351 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:39:14.557424 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:14.557756 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:14.557768 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:14.557778 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 04:39:14.557924 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 04:39:14.558214 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":174,"threads_started":1}
I20260808 04:39:14.560809 10319 heartbeater.cc:347] Connected to a master server at 127.15.228.126:41373
I20260808 04:39:14.561357 10319 heartbeater.cc:464] Registering TS with master...
I20260808 04:39:14.562136 10319 heartbeater.cc:511] Master 127.15.228.126:41373 requested a full tablet report, sending...
I20260808 04:39:14.564256 10061 ts_manager.cc:194] Registered new tserver with Master: c5820b6dfc3743bda5ed96b5218387fd (127.15.228.65:36937)
I20260808 04:39:14.566110 10061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.228.65:47985
W20260808 04:39:14.698887 10323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:14.699409 10323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:14.699465 10323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:14.706398 10323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:14.706524 10323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:14.706601 10323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:39:14.714277 10323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:41373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:14.715602 10323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:14.718295 10323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:14.730198 10336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:14.730223 10333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:14.730245 10331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:14.730818 10323 server_base.cc:1034] running on GCE node
I20260808 04:39:14.731335 10323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:14.732043 10323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:14.735391 10323 hybrid_clock.cc:648] HybridClock initialized: now 1786163954735328 us; error 158 us; skew 500 ppm
I20260808 04:39:14.738369 10323 webserver.cc:460] Webserver started at http://127.15.228.66:33501/ using document root <none> and password file <none>
I20260808 04:39:14.739122 10323 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:14.739205 10323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:14.739743 10323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:14.743526 10323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "7f4d2507b7334fb4bc74fd7cfa68ef35"
format_stamp: "Formatted at 2026-08-08 04:39:14 on dist-test-slave-drl0"
server_key: "63075544132b988386b608fee5f2e3f6"
server_key_iv: "f9e8b84dd55b9f260d67182cdec87555"
server_key_version: "encryptionkey@0"
I20260808 04:39:14.744251 10323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "7f4d2507b7334fb4bc74fd7cfa68ef35"
format_stamp: "Formatted at 2026-08-08 04:39:14 on dist-test-slave-drl0"
server_key: "63075544132b988386b608fee5f2e3f6"
server_key_iv: "f9e8b84dd55b9f260d67182cdec87555"
server_key_version: "encryptionkey@0"
I20260808 04:39:14.748600 10323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:39:14.751422 10343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:14.752942 10323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:39:14.753118 10323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "7f4d2507b7334fb4bc74fd7cfa68ef35"
format_stamp: "Formatted at 2026-08-08 04:39:14 on dist-test-slave-drl0"
server_key: "63075544132b988386b608fee5f2e3f6"
server_key_iv: "f9e8b84dd55b9f260d67182cdec87555"
server_key_version: "encryptionkey@0"
I20260808 04:39:14.753260 10323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:14.767046 10323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:14.768456 10323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:14.768669 10323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:14.769342 10323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:39:14.770622 10323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:39:14.770679 10323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:14.770774 10323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:39:14.770818 10323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:14.784348 10323 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:38095
I20260808 04:39:14.784368 10501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:38095 every 8 connection(s)
I20260808 04:39:14.785725 10323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:39:14.787915 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 10323
I20260808 04:39:14.788074 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:39:14.788494 16273 external_mini_cluster.cc:1428] Setting key 492d7f6e3901b2a9ac9c22d4cfd8c9dc
I20260808 04:39:14.791002 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:41373
--builtin_ntp_servers=127.15.228.84:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:39:14.798774 10116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:14.788479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:40987 (local address 127.15.228.126:41373)
0808 04:39:14.788651 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:14.788658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:14.790855 (+ 2197us) server_negotiation.cc:407] Connection header received
0808 04:39:14.793145 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:14.793148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:14.793214 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:14.793349 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:14.794412 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:14.795080 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:14.796392 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:14.796637 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:14.796763 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:14.798171 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:14.798183 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:14.798186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:14.798200 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:14.798220 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:14.798228 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:14.798334 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:14.798539 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 04:39:14.798607 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 04:39:14.799906 10507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:14.788994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:41373 (local address 127.15.228.66:40987)
0808 04:39:14.790555 (+ 1561us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:14.790662 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:14.792852 (+ 2190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:14.793565 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:14.793573 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:14.793674 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:14.794247 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:14.794261 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:14.795220 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:14.795224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:14.796234 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:14.796245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:14.796816 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:14.796821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:14.797011 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:14.797774 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:39:14.797776 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:14.797829 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:39:14.797902 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:14.798342 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:14.798354 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:14.798360 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 04:39:14.798540 (+ 180us) client_negotiation.cc:240] Negotiation successful
0808 04:39:14.798993 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":256,"threads_started":1}
I20260808 04:39:14.801815 10504 heartbeater.cc:347] Connected to a master server at 127.15.228.126:41373
I20260808 04:39:14.802420 10504 heartbeater.cc:464] Registering TS with master...
I20260808 04:39:14.803156 10504 heartbeater.cc:511] Master 127.15.228.126:41373 requested a full tablet report, sending...
I20260808 04:39:14.804678 10061 ts_manager.cc:194] Registered new tserver with Master: 7f4d2507b7334fb4bc74fd7cfa68ef35 (127.15.228.66:38095)
I20260808 04:39:14.805640 10061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.228.66:40987
W20260808 04:39:14.944078 10509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:14.944602 10509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:14.944700 10509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:14.950847 10509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:14.951064 10509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:14.951359 10509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:39:14.959107 10509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:41373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:14.960539 10509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:14.962229 10509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:14.973644 10518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:14.973644 10515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:14.974031 10516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:14.975598 10509 server_base.cc:1034] running on GCE node
I20260808 04:39:14.976944 10509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:14.977986 10509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:14.979504 10509 hybrid_clock.cc:648] HybridClock initialized: now 1786163954979453 us; error 71 us; skew 500 ppm
I20260808 04:39:14.982511 10509 webserver.cc:460] Webserver started at http://127.15.228.67:44191/ using document root <none> and password file <none>
I20260808 04:39:14.983418 10509 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:14.983536 10509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:14.983841 10509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:14.986284 10509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "b57a6395306f45e3b4cb711091b142b1"
format_stamp: "Formatted at 2026-08-08 04:39:14 on dist-test-slave-drl0"
server_key: "d4660219c6e3a789b8e8416d5ddb2687"
server_key_iv: "c9a0a0381f316c1cf9a63a137c009323"
server_key_version: "encryptionkey@0"
I20260808 04:39:14.987058 10509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "b57a6395306f45e3b4cb711091b142b1"
format_stamp: "Formatted at 2026-08-08 04:39:14 on dist-test-slave-drl0"
server_key: "d4660219c6e3a789b8e8416d5ddb2687"
server_key_iv: "c9a0a0381f316c1cf9a63a137c009323"
server_key_version: "encryptionkey@0"
I20260808 04:39:14.991415 10509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:39:14.994400 10526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:14.997025 10509 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:39:14.997308 10509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "b57a6395306f45e3b4cb711091b142b1"
format_stamp: "Formatted at 2026-08-08 04:39:14 on dist-test-slave-drl0"
server_key: "d4660219c6e3a789b8e8416d5ddb2687"
server_key_iv: "c9a0a0381f316c1cf9a63a137c009323"
server_key_version: "encryptionkey@0"
I20260808 04:39:14.997498 10509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:15.030916 10509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:15.032390 10509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:15.032867 10509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:15.034040 10509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:39:15.038147 10509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:39:15.038237 10509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:15.039357 10509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:39:15.039456 10509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:15.061079 10509 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:33313
I20260808 04:39:15.061107 10671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:33313 every 8 connection(s)
I20260808 04:39:15.062515 10509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:39:15.064419 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 10509
I20260808 04:39:15.064584 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:39:15.064968 16273 external_mini_cluster.cc:1428] Setting key fe4c2833ecc98da392c26b4777f10cad
I20260808 04:39:15.074847 10116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:15.064976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:41971 (local address 127.15.228.126:41373)
0808 04:39:15.065160 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:15.065164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:15.066578 (+ 1414us) server_negotiation.cc:407] Connection header received
0808 04:39:15.068793 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:15.068797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:15.068873 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:15.068968 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:15.070215 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.070816 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.072229 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.072469 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.072595 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:15.074195 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:15.074199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:15.074202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:15.074218 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:15.074245 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:15.074255 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:15.074367 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:15.074570 (+ 203us) server_negotiation.cc:299] Negotiation successful
0808 04:39:15.074670 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 04:39:15.076067 10674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:15.065378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:41373 (local address 127.15.228.67:41971)
0808 04:39:15.066356 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:15.066423 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:15.068520 (+ 2097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:15.069239 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:15.069249 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:15.069365 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:15.070041 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:15.070056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.071014 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.071018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:15.072046 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:15.072059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.072667 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.072672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:15.072887 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:15.073757 (+ 870us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:39:15.073760 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:15.073827 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:39:15.073930 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:15.074375 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:15.074389 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:15.074398 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 04:39:15.074557 (+ 159us) client_negotiation.cc:240] Negotiation successful
0808 04:39:15.075015 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":195,"threads_started":1}
I20260808 04:39:15.078030 10672 heartbeater.cc:347] Connected to a master server at 127.15.228.126:41373
I20260808 04:39:15.078655 10672 heartbeater.cc:464] Registering TS with master...
I20260808 04:39:15.079952 10672 heartbeater.cc:511] Master 127.15.228.126:41373 requested a full tablet report, sending...
I20260808 04:39:15.081610 10061 ts_manager.cc:194] Registered new tserver with Master: b57a6395306f45e3b4cb711091b142b1 (127.15.228.67:33313)
I20260808 04:39:15.082397 10061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.228.67:41971
I20260808 04:39:15.093887 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:39:15.102979 10116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:15.096934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.15.228.126:41373)
0808 04:39:15.097086 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:15.097099 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:15.097547 (+ 448us) server_negotiation.cc:407] Connection header received
0808 04:39:15.097678 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:15.097683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:15.097749 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:15.097841 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:15.098443 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.099223 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.100389 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.100787 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.100954 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:15.102260 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:15.102266 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:15.102270 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:15.102291 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:15.102319 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:15.102334 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:15.102491 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:15.102641 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 04:39:15.102692 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:39:15.107148 10061 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:39:15.113656 10061 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:39:15.134325 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:15.127439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:36937 (local address 127.0.0.1:40602)
0808 04:39:15.128142 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:15.128197 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:15.128349 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:15.128993 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:15.129000 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:15.129023 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:15.129647 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:15.129657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.131133 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.131137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:15.132648 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:15.132660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.133898 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.133903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:15.133993 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:15.134032 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 04:39:15.134097 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 04:39:15.134169 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":65,"thread_start_us":126,"threads_started":1}
I20260808 04:39:15.134712 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:15.127614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40602 (local address 127.15.228.65:36937)
0808 04:39:15.128291 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:15.128299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:15.128327 (+ 28us) server_negotiation.cc:407] Connection header received
0808 04:39:15.128608 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:15.128616 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:15.128835 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:15.128991 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:15.129792 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.131007 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.132897 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.133641 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.134212 (+ 571us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:15.134407 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:15.134474 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 04:39:15.134581 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":126,"threads_started":1}
I20260808 04:39:15.135082 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:15.128266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:38095 (local address 127.0.0.1:53782)
0808 04:39:15.128549 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:15.128563 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:15.128648 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:15.129772 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:15.129775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:15.129795 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:15.129915 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:15.129923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.131520 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.131525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:15.132510 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:15.132520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.133616 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.133620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:15.133742 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:15.133823 (+ 81us) client_negotiation.cc:769] Sending connection context
0808 04:39:15.133951 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 04:39:15.134100 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":77,"thread_start_us":119,"threads_started":1}
I20260808 04:39:15.135079 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:15.128562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53782 (local address 127.15.228.66:38095)
0808 04:39:15.129306 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:15.129317 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:15.129341 (+ 24us) server_negotiation.cc:407] Connection header received
0808 04:39:15.129427 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:15.129433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:15.129613 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:15.129788 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:15.130054 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.131391 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.132647 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.133457 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.133580 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:15.133810 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:15.134758 (+ 948us) server_negotiation.cc:299] Negotiation successful
0808 04:39:15.134882 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":115,"threads_started":1}
I20260808 04:39:15.135078 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:15.129208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.15.228.67:33313)
0808 04:39:15.129610 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:15.129619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:15.129638 (+ 19us) server_negotiation.cc:407] Connection header received
0808 04:39:15.129698 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:15.129702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:15.129900 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:15.130049 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:15.130285 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.131412 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.133048 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.133728 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.133847 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:15.134040 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:15.134654 (+ 614us) server_negotiation.cc:299] Negotiation successful
0808 04:39:15.134770 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":197,"threads_started":1}
I20260808 04:39:15.134545 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:15.128595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:33313 (local address 127.0.0.1:60508)
0808 04:39:15.129140 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:15.129157 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:15.129252 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:15.130064 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:15.130067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:15.130082 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:15.130157 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:15.130166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.131551 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.131555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:15.132910 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:15.132919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.134108 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.134112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:15.134199 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:15.134223 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:39:15.134312 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 04:39:15.134379 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":37,"thread_start_us":92,"threads_started":1}
I20260808 04:39:15.139091 10226 tablet_service.cc:1467] Processing CreateTablet for tablet 29be8fa8cf524d5ba46851e8bd1b99b5 (DEFAULT_TABLE table=test-table [id=e41fa0d249674522833bfae75bfc1ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:39:15.139158 10409 tablet_service.cc:1467] Processing CreateTablet for tablet 29be8fa8cf524d5ba46851e8bd1b99b5 (DEFAULT_TABLE table=test-table [id=e41fa0d249674522833bfae75bfc1ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:39:15.139639 10594 tablet_service.cc:1467] Processing CreateTablet for tablet 29be8fa8cf524d5ba46851e8bd1b99b5 (DEFAULT_TABLE table=test-table [id=e41fa0d249674522833bfae75bfc1ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:39:15.140465 10226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29be8fa8cf524d5ba46851e8bd1b99b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:15.140470 10409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29be8fa8cf524d5ba46851e8bd1b99b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:15.140543 10594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29be8fa8cf524d5ba46851e8bd1b99b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:15.149055 10694 tablet_bootstrap.cc:492] T 29be8fa8cf524d5ba46851e8bd1b99b5 P c5820b6dfc3743bda5ed96b5218387fd: Bootstrap starting.
I20260808 04:39:15.149888 10695 tablet_bootstrap.cc:492] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1: Bootstrap starting.
I20260808 04:39:15.151607 10696 tablet_bootstrap.cc:492] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35: Bootstrap starting.
I20260808 04:39:15.152232 10694 tablet_bootstrap.cc:654] T 29be8fa8cf524d5ba46851e8bd1b99b5 P c5820b6dfc3743bda5ed96b5218387fd: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:15.152873 10695 tablet_bootstrap.cc:654] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:15.153680 10694 log.cc:824] T 29be8fa8cf524d5ba46851e8bd1b99b5 P c5820b6dfc3743bda5ed96b5218387fd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:15.153858 10695 log.cc:824] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:15.154150 10696 tablet_bootstrap.cc:654] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:15.154985 10696 log.cc:824] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:15.156508 10694 tablet_bootstrap.cc:492] T 29be8fa8cf524d5ba46851e8bd1b99b5 P c5820b6dfc3743bda5ed96b5218387fd: No bootstrap required, opened a new log
I20260808 04:39:15.156756 10694 ts_tablet_manager.cc:1397] T 29be8fa8cf524d5ba46851e8bd1b99b5 P c5820b6dfc3743bda5ed96b5218387fd: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260808 04:39:15.157729 10696 tablet_bootstrap.cc:492] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35: No bootstrap required, opened a new log
I20260808 04:39:15.157935 10696 ts_tablet_manager.cc:1397] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260808 04:39:15.159572 10695 tablet_bootstrap.cc:492] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1: No bootstrap required, opened a new log
I20260808 04:39:15.159804 10695 ts_tablet_manager.cc:1397] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260808 04:39:15.162263 10694 raft_consensus.cc:348] T 29be8fa8cf524d5ba46851e8bd1b99b5 P c5820b6dfc3743bda5ed96b5218387fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820b6dfc3743bda5ed96b5218387fd" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36937 } } peers { permanent_uuid: "7f4d2507b7334fb4bc74fd7cfa68ef35" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 38095 } } peers { permanent_uuid: "b57a6395306f45e3b4cb711091b142b1" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 33313 } }
I20260808 04:39:15.162413 10696 raft_consensus.cc:348] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820b6dfc3743bda5ed96b5218387fd" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36937 } } peers { permanent_uuid: "7f4d2507b7334fb4bc74fd7cfa68ef35" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 38095 } } peers { permanent_uuid: "b57a6395306f45e3b4cb711091b142b1" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 33313 } }
I20260808 04:39:15.162706 10694 raft_consensus.cc:374] T 29be8fa8cf524d5ba46851e8bd1b99b5 P c5820b6dfc3743bda5ed96b5218387fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:15.162863 10694 raft_consensus.cc:729] T 29be8fa8cf524d5ba46851e8bd1b99b5 P c5820b6dfc3743bda5ed96b5218387fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5820b6dfc3743bda5ed96b5218387fd, State: Initialized, Role: FOLLOWER
I20260808 04:39:15.162882 10696 raft_consensus.cc:374] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:15.162989 10696 raft_consensus.cc:729] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f4d2507b7334fb4bc74fd7cfa68ef35, State: Initialized, Role: FOLLOWER
I20260808 04:39:15.163630 10696 consensus_queue.cc:260] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820b6dfc3743bda5ed96b5218387fd" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36937 } } peers { permanent_uuid: "7f4d2507b7334fb4bc74fd7cfa68ef35" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 38095 } } peers { permanent_uuid: "b57a6395306f45e3b4cb711091b142b1" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 33313 } }
I20260808 04:39:15.163630 10694 consensus_queue.cc:260] T 29be8fa8cf524d5ba46851e8bd1b99b5 P c5820b6dfc3743bda5ed96b5218387fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820b6dfc3743bda5ed96b5218387fd" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36937 } } peers { permanent_uuid: "7f4d2507b7334fb4bc74fd7cfa68ef35" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 38095 } } peers { permanent_uuid: "b57a6395306f45e3b4cb711091b142b1" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 33313 } }
I20260808 04:39:15.163980 10695 raft_consensus.cc:348] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820b6dfc3743bda5ed96b5218387fd" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36937 } } peers { permanent_uuid: "7f4d2507b7334fb4bc74fd7cfa68ef35" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 38095 } } peers { permanent_uuid: "b57a6395306f45e3b4cb711091b142b1" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 33313 } }
I20260808 04:39:15.164266 10695 raft_consensus.cc:374] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:15.164352 10695 raft_consensus.cc:729] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57a6395306f45e3b4cb711091b142b1, State: Initialized, Role: FOLLOWER
I20260808 04:39:15.164908 10319 heartbeater.cc:503] Master 127.15.228.126:41373 was elected leader, sending a full tablet report...
I20260808 04:39:15.165088 10504 heartbeater.cc:503] Master 127.15.228.126:41373 was elected leader, sending a full tablet report...
I20260808 04:39:15.165303 10694 ts_tablet_manager.cc:1428] T 29be8fa8cf524d5ba46851e8bd1b99b5 P c5820b6dfc3743bda5ed96b5218387fd: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260808 04:39:15.165782 10696 ts_tablet_manager.cc:1428] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260808 04:39:15.166319 10695 consensus_queue.cc:260] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820b6dfc3743bda5ed96b5218387fd" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36937 } } peers { permanent_uuid: "7f4d2507b7334fb4bc74fd7cfa68ef35" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 38095 } } peers { permanent_uuid: "b57a6395306f45e3b4cb711091b142b1" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 33313 } }
I20260808 04:39:15.167544 10695 ts_tablet_manager.cc:1428] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260808 04:39:15.167912 10672 heartbeater.cc:503] Master 127.15.228.126:41373 was elected leader, sending a full tablet report...
I20260808 04:39:15.238647 10708 raft_consensus.cc:482] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:39:15.238972 10708 raft_consensus.cc:504] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820b6dfc3743bda5ed96b5218387fd" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36937 } } peers { permanent_uuid: "7f4d2507b7334fb4bc74fd7cfa68ef35" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 38095 } } peers { permanent_uuid: "b57a6395306f45e3b4cb711091b142b1" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 33313 } }
I20260808 04:39:15.240329 10708 leader_election.cc:288] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5820b6dfc3743bda5ed96b5218387fd (127.15.228.65:36937), 7f4d2507b7334fb4bc74fd7cfa68ef35 (127.15.228.66:38095)
I20260808 04:39:15.246039 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:15.241215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:51433 (local address 127.15.228.66:38095)
0808 04:39:15.241498 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:15.241505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:15.241523 (+ 18us) server_negotiation.cc:407] Connection header received
0808 04:39:15.241584 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:15.241586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:15.241634 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:15.241738 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:15.242250 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.243095 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.244567 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.245175 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.245323 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:15.245379 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:15.245818 (+ 439us) server_negotiation.cc:299] Negotiation successful
0808 04:39:15.245902 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260808 04:39:15.246037 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:15.240921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:50715 (local address 127.15.228.65:36937)
0808 04:39:15.241246 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:15.241252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:15.241273 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:39:15.241337 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:15.241342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:15.241427 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:15.241532 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:15.241912 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.242814 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.244550 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.245158 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.245306 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:15.245364 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:15.245816 (+ 452us) server_negotiation.cc:299] Negotiation successful
0808 04:39:15.245914 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260808 04:39:15.246059 10674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:15.240569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:36937 (local address 127.15.228.67:50715)
0808 04:39:15.240726 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:15.240745 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:15.240849 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:15.241582 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:15.241586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:15.241607 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:15.241754 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:15.241763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.243016 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.243018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:15.244366 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:15.244373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.245341 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.245345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:15.245588 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:15.245668 (+ 80us) client_negotiation.cc:769] Sending connection context
0808 04:39:15.245760 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 04:39:15.245877 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":164}
I20260808 04:39:15.246068 10709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:15.240885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:38095 (local address 127.15.228.67:51433)
0808 04:39:15.241319 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:15.241332 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:15.241428 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:15.241930 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:15.241933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:15.241958 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:15.242128 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:15.242134 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.243192 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.243195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:15.244361 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:15.244373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.245341 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.245345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:15.245581 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:15.245671 (+ 90us) client_negotiation.cc:769] Sending connection context
0808 04:39:15.245787 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 04:39:15.245877 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":133,"thread_start_us":175,"threads_started":1}
I20260808 04:39:15.247259 10250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29be8fa8cf524d5ba46851e8bd1b99b5" candidate_uuid: "b57a6395306f45e3b4cb711091b142b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5820b6dfc3743bda5ed96b5218387fd" is_pre_election: true
I20260808 04:39:15.247253 10436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29be8fa8cf524d5ba46851e8bd1b99b5" candidate_uuid: "b57a6395306f45e3b4cb711091b142b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f4d2507b7334fb4bc74fd7cfa68ef35" is_pre_election: true
I20260808 04:39:15.247601 10250 raft_consensus.cc:2454] T 29be8fa8cf524d5ba46851e8bd1b99b5 P c5820b6dfc3743bda5ed96b5218387fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b57a6395306f45e3b4cb711091b142b1 in term 0.
I20260808 04:39:15.247617 10436 raft_consensus.cc:2454] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b57a6395306f45e3b4cb711091b142b1 in term 0.
I20260808 04:39:15.248185 10529 leader_election.cc:302] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f4d2507b7334fb4bc74fd7cfa68ef35, b57a6395306f45e3b4cb711091b142b1; no voters:
I20260808 04:39:15.248562 10708 raft_consensus.cc:2781] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:39:15.248682 10708 raft_consensus.cc:482] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:39:15.248791 10708 raft_consensus.cc:3037] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:15.250034 10708 raft_consensus.cc:504] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820b6dfc3743bda5ed96b5218387fd" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36937 } } peers { permanent_uuid: "7f4d2507b7334fb4bc74fd7cfa68ef35" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 38095 } } peers { permanent_uuid: "b57a6395306f45e3b4cb711091b142b1" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 33313 } }
I20260808 04:39:15.250413 10708 leader_election.cc:288] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [CANDIDATE]: Term 1 election: Requested vote from peers c5820b6dfc3743bda5ed96b5218387fd (127.15.228.65:36937), 7f4d2507b7334fb4bc74fd7cfa68ef35 (127.15.228.66:38095)
I20260808 04:39:15.251341 10250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29be8fa8cf524d5ba46851e8bd1b99b5" candidate_uuid: "b57a6395306f45e3b4cb711091b142b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5820b6dfc3743bda5ed96b5218387fd"
I20260808 04:39:15.251394 10436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29be8fa8cf524d5ba46851e8bd1b99b5" candidate_uuid: "b57a6395306f45e3b4cb711091b142b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f4d2507b7334fb4bc74fd7cfa68ef35"
I20260808 04:39:15.251515 10250 raft_consensus.cc:3037] T 29be8fa8cf524d5ba46851e8bd1b99b5 P c5820b6dfc3743bda5ed96b5218387fd [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:15.251559 10436 raft_consensus.cc:3037] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:15.252894 10436 raft_consensus.cc:2454] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b57a6395306f45e3b4cb711091b142b1 in term 1.
I20260808 04:39:15.252892 10250 raft_consensus.cc:2454] T 29be8fa8cf524d5ba46851e8bd1b99b5 P c5820b6dfc3743bda5ed96b5218387fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b57a6395306f45e3b4cb711091b142b1 in term 1.
I20260808 04:39:15.253284 10529 leader_election.cc:302] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b57a6395306f45e3b4cb711091b142b1, c5820b6dfc3743bda5ed96b5218387fd; no voters:
I20260808 04:39:15.253480 10708 raft_consensus.cc:2781] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:15.253726 10708 raft_consensus.cc:686] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [term 1 LEADER]: Becoming Leader. State: Replica: b57a6395306f45e3b4cb711091b142b1, State: Running, Role: LEADER
I20260808 04:39:15.254133 10708 consensus_queue.cc:237] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820b6dfc3743bda5ed96b5218387fd" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36937 } } peers { permanent_uuid: "7f4d2507b7334fb4bc74fd7cfa68ef35" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 38095 } } peers { permanent_uuid: "b57a6395306f45e3b4cb711091b142b1" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 33313 } }
I20260808 04:39:15.258432 10061 catalog_manager.cc:5495] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to b57a6395306f45e3b4cb711091b142b1 (127.15.228.67). New cstate: current_term: 1 leader_uuid: "b57a6395306f45e3b4cb711091b142b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5820b6dfc3743bda5ed96b5218387fd" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f4d2507b7334fb4bc74fd7cfa68ef35" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 38095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b57a6395306f45e3b4cb711091b142b1" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 33313 } health_report { overall_health: HEALTHY } } }
W20260808 04:39:15.287710 10505 tablet.cc:2369] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:39:15.293084 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:15.288361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60514 (local address 127.15.228.67:33313)
0808 04:39:15.288518 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:15.288521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:15.288535 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:39:15.288606 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:15.288608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:15.288665 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:15.288763 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:39:15.289194 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.289813 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.290848 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:15.291291 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:15.291427 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:15.291799 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:39:15.292412 (+ 613us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:39:15.292564 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:15.292828 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 04:39:15.292879 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260808 04:39:15.298591 10320 tablet.cc:2369] T 29be8fa8cf524d5ba46851e8bd1b99b5 P c5820b6dfc3743bda5ed96b5218387fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:39:15.301465 10250 raft_consensus.cc:1261] T 29be8fa8cf524d5ba46851e8bd1b99b5 P c5820b6dfc3743bda5ed96b5218387fd [term 1 FOLLOWER]: Refusing update from remote peer b57a6395306f45e3b4cb711091b142b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:39:15.301666 10436 raft_consensus.cc:1261] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35 [term 1 FOLLOWER]: Refusing update from remote peer b57a6395306f45e3b4cb711091b142b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:39:15.302580 10710 consensus_queue.cc:1035] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5820b6dfc3743bda5ed96b5218387fd" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:39:15.302884 10708 consensus_queue.cc:1035] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f4d2507b7334fb4bc74fd7cfa68ef35" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 38095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:39:15.314949 10715 mvcc.cc:205] Tried to move back new op lower bound from 7316127560904101888 to 7316127560724332544. Current Snapshot: MvccSnapshot[applied={T|T < 7316127560904101888}]
I20260808 04:39:15.321772 10058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:39:15.321946 10058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:39:15.324890 10058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22cf737d47e0491c9ccf2d8ebd0da51e: Sending DeleteTablet for 3 replicas of tablet 29be8fa8cf524d5ba46851e8bd1b99b5
I20260808 04:39:15.325855 10226 tablet_service.cc:1514] Processing DeleteTablet for tablet 29be8fa8cf524d5ba46851e8bd1b99b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:39:15 UTC) from {username='slave'} at 127.0.0.1:40602
I20260808 04:39:15.325903 10409 tablet_service.cc:1514] Processing DeleteTablet for tablet 29be8fa8cf524d5ba46851e8bd1b99b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:39:15 UTC) from {username='slave'} at 127.0.0.1:53782
I20260808 04:39:15.326622 10724 tablet_replica.cc:330] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35: stopping tablet replica
I20260808 04:39:15.327149 10723 tablet_replica.cc:330] T 29be8fa8cf524d5ba46851e8bd1b99b5 P c5820b6dfc3743bda5ed96b5218387fd: stopping tablet replica
I20260808 04:39:15.327107 10594 tablet_service.cc:1514] Processing DeleteTablet for tablet 29be8fa8cf524d5ba46851e8bd1b99b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:39:15 UTC) from {username='slave'} at 127.0.0.1:60508
I20260808 04:39:15.327389 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 10140
I20260808 04:39:15.327643 10723 raft_consensus.cc:2229] T 29be8fa8cf524d5ba46851e8bd1b99b5 P c5820b6dfc3743bda5ed96b5218387fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:39:15.329845 10724 raft_consensus.cc:2229] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:39:15.330276 10724 raft_consensus.cc:2258] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:39:15.333428 10724 ts_tablet_manager.cc:1905] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:39:15.336275 10724 ts_tablet_manager.cc:1918] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:39:15.336452 10724 log.cc:1196] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/29be8fa8cf524d5ba46851e8bd1b99b5
I20260808 04:39:15.336848 10724 ts_tablet_manager.cc:1939] T 29be8fa8cf524d5ba46851e8bd1b99b5 P 7f4d2507b7334fb4bc74fd7cfa68ef35: Deleting consensus metadata
I20260808 04:39:15.342761 10043 catalog_manager.cc:4841] TS 7f4d2507b7334fb4bc74fd7cfa68ef35 (127.15.228.66:38095): tablet 29be8fa8cf524d5ba46851e8bd1b99b5 (table test-table [id=e41fa0d249674522833bfae75bfc1ca1]) successfully deleted
I20260808 04:39:15.343361 10725 tablet_replica.cc:330] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1: stopping tablet replica
I20260808 04:39:15.343863 10725 raft_consensus.cc:2229] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:39:15.344305 10725 raft_consensus.cc:2258] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 04:39:15.349839 10043 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.228.65:36937 (error 108)
I20260808 04:39:15.349982 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 10323
I20260808 04:39:15.351269 10725 ts_tablet_manager.cc:1905] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:39:15.354074 10725 ts_tablet_manager.cc:1918] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:39:15.354177 10725 log.cc:1196] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/29be8fa8cf524d5ba46851e8bd1b99b5
I20260808 04:39:15.354467 10725 ts_tablet_manager.cc:1939] T 29be8fa8cf524d5ba46851e8bd1b99b5 P b57a6395306f45e3b4cb711091b142b1: Deleting consensus metadata
I20260808 04:39:15.354660 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:15.354124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:36937 (local address 127.0.0.1:40608)
0808 04:39:15.354430 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:15.354533 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.65:36937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
W20260808 04:39:15.355098 10043 catalog_manager.cc:4584] TS c5820b6dfc3743bda5ed96b5218387fd (127.15.228.65:36937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 29be8fa8cf524d5ba46851e8bd1b99b5: Network error: Client connection negotiation failed: client connection to 127.15.228.65:36937: connect: Connection refused (error 111)
I20260808 04:39:15.356437 10043 catalog_manager.cc:4841] TS b57a6395306f45e3b4cb711091b142b1 (127.15.228.67:33313): tablet 29be8fa8cf524d5ba46851e8bd1b99b5 (table test-table [id=e41fa0d249674522833bfae75bfc1ca1]) successfully deleted
I20260808 04:39:15.362138 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 10509
I20260808 04:39:15.375867 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 10022
2026-08-08T04:39:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1438 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:15 dist-test-slave-drl0 krb5kdc[10730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:15 dist-test-slave-drl0 krb5kdc[10730](info): set up 2 sockets
Aug 08 04:39:15 dist-test-slave-drl0 krb5kdc[10730](info): commencing operation
krb5kdc: starting...
W20260808 04:39:16.682977 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.276s user 0.000s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:16 dist-test-slave-drl0 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:39:16.845448 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:33963
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:38685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:33963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:16.992532 10748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:16.992859 10748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:16.992935 10748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:16.997850 10748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:16.998023 10748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:16.998070 10748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:16.998112 10748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:16.998171 10748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:17.005494 10748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:38685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:33963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:33963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:17.006949 10748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:17.008106 10748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:17.018227 10757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:17.018256 10754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:17.018256 10753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:17.018324 10748 server_base.cc:1034] running on GCE node
I20260808 04:39:17.019462 10748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:17.021564 10748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:17.022799 10748 hybrid_clock.cc:648] HybridClock initialized: now 1786163957022755 us; error 61 us; skew 500 ppm
Aug 08 04:39:17 dist-test-slave-drl0 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163957, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:17.025892 10748 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:39:17.026845 10748 webserver.cc:460] Webserver started at http://127.15.228.126:33511/ using document root <none> and password file <none>
I20260808 04:39:17.027479 10748 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:17.027581 10748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:17.027834 10748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:17.030015 10748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "bd04f6c991b44fa38839ba47725bdc10"
format_stamp: "Formatted at 2026-08-08 04:39:17 on dist-test-slave-drl0"
server_key: "04acb68bfb9bfc239a56aacfe42976d2"
server_key_iv: "e2db902fec4dd2f244e87ec548d56e02"
server_key_version: "encryptionkey@0"
I20260808 04:39:17.030604 10748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "bd04f6c991b44fa38839ba47725bdc10"
format_stamp: "Formatted at 2026-08-08 04:39:17 on dist-test-slave-drl0"
server_key: "04acb68bfb9bfc239a56aacfe42976d2"
server_key_iv: "e2db902fec4dd2f244e87ec548d56e02"
server_key_version: "encryptionkey@0"
I20260808 04:39:17.034904 10748 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260808 04:39:17.038640 10766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:17.040874 10748 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:39:17.041267 10748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "bd04f6c991b44fa38839ba47725bdc10"
format_stamp: "Formatted at 2026-08-08 04:39:17 on dist-test-slave-drl0"
server_key: "04acb68bfb9bfc239a56aacfe42976d2"
server_key_iv: "e2db902fec4dd2f244e87ec548d56e02"
server_key_version: "encryptionkey@0"
I20260808 04:39:17.041460 10748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:17.063207 10748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:17.067481 10748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:17.067734 10748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:17.077008 10748 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:33963
I20260808 04:39:17.077019 10838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:33963 every 8 connection(s)
I20260808 04:39:17.078290 10748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:17.082784 10841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:17.086024 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 10748
I20260808 04:39:17.086223 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:17.086808 16273 external_mini_cluster.cc:1428] Setting key 2e869ca1d1b1d609b07c80e5ce035cf8
I20260808 04:39:17.092036 10841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10: Bootstrap starting.
I20260808 04:39:17.094501 10841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:17.095378 10841 log.cc:824] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:17.098073 10841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10: No bootstrap required, opened a new log
Aug 08 04:39:17 dist-test-slave-drl0 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:39:17.103413 10841 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd04f6c991b44fa38839ba47725bdc10" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33963 } }
I20260808 04:39:17.103721 10841 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:17.103796 10841 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd04f6c991b44fa38839ba47725bdc10, State: Initialized, Role: FOLLOWER
I20260808 04:39:17.104378 10841 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd04f6c991b44fa38839ba47725bdc10" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33963 } }
I20260808 04:39:17.104559 10841 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:17.104631 10841 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:17.104729 10841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:17.105595 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:17.088581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38342 (local address 127.15.228.126:33963)
0808 04:39:17.089099 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:17.089115 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:17.089156 (+ 41us) server_negotiation.cc:407] Connection header received
0808 04:39:17.089938 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:17.089959 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:17.090300 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:17.090671 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:17.091962 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:17.093368 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:17.094535 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:17.095367 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:17.095588 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:17.098979 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:17.098988 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:17.099007 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:17.099046 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:17.101695 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:17.102282 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:17.102288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:17.102295 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:17.102567 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:17.102987 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:17.102991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:17.102992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:17.103634 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:17.103882 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:17.104210 (+ 328us) server_negotiation.cc:299] Negotiation successful
0808 04:39:17.104535 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":164,"threads_started":1}
I20260808 04:39:17.105983 10841 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd04f6c991b44fa38839ba47725bdc10" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33963 } }
I20260808 04:39:17.106360 10841 leader_election.cc:302] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd04f6c991b44fa38839ba47725bdc10; no voters:
I20260808 04:39:17.106730 10841 leader_election.cc:288] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:17.106882 10848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:17.107167 10848 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10 [term 1 LEADER]: Becoming Leader. State: Replica: bd04f6c991b44fa38839ba47725bdc10, State: Running, Role: LEADER
I20260808 04:39:17.107540 10848 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd04f6c991b44fa38839ba47725bdc10" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33963 } }
I20260808 04:39:17.108104 10841 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:17.108762 10848 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd04f6c991b44fa38839ba47725bdc10. Latest consensus state: current_term: 1 leader_uuid: "bd04f6c991b44fa38839ba47725bdc10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd04f6c991b44fa38839ba47725bdc10" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33963 } } }
I20260808 04:39:17.108975 10848 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:17.109220 10850 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd04f6c991b44fa38839ba47725bdc10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd04f6c991b44fa38839ba47725bdc10" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 33963 } } }
I20260808 04:39:17.109354 10850 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:17.109535 10861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:17.112958 10861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:17.120432 10861 catalog_manager.cc:1347] Generated new cluster ID: 2572811c03c642aea87a6581a10b1420
I20260808 04:39:17.120579 10861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:17.128171 10861 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:17.129487 10861 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:17.137691 10861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10: Generated new TSK 0
I20260808 04:39:17.139042 10861 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.228.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
Entry for principal kudu/127.15.228.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab.
I20260808 04:39:17.180902 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33963
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:38685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:17.338083 10879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:17.338488 10879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:17.338555 10879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:17.344933 10879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:17.345042 10879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:17.345181 10879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:39:17.352722 10879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:38685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.65.keytab
--principal=kudu/127.15.228.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:17.354588 10879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:17.356053 10879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:17.370152 10886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:17.370153 10888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:17.370322 10884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:17.370714 10879 server_base.cc:1034] running on GCE node
I20260808 04:39:17.371771 10879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:17.373240 10879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:17.374686 10879 hybrid_clock.cc:648] HybridClock initialized: now 1786163957374622 us; error 52 us; skew 500 ppm
Aug 08 04:39:17 dist-test-slave-drl0 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163957, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:17.379650 10879 init.cc:377] Logged in from keytab as kudu/127.15.228.65@KRBTEST.COM (short username kudu)
I20260808 04:39:17.381198 10879 webserver.cc:460] Webserver started at http://127.15.228.65:43655/ using document root <none> and password file <none>
I20260808 04:39:17.382038 10879 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:17.382166 10879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:17.382475 10879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:17.385687 10879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "aa67ef66734840c1a19f526b42be84fb"
format_stamp: "Formatted at 2026-08-08 04:39:17 on dist-test-slave-drl0"
server_key: "9546819bd5e0d78288f6b3ab125d91a2"
server_key_iv: "22e1fd2443a5d49f326fb647694ec7d9"
server_key_version: "encryptionkey@0"
I20260808 04:39:17.386392 10879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "aa67ef66734840c1a19f526b42be84fb"
format_stamp: "Formatted at 2026-08-08 04:39:17 on dist-test-slave-drl0"
server_key: "9546819bd5e0d78288f6b3ab125d91a2"
server_key_iv: "22e1fd2443a5d49f326fb647694ec7d9"
server_key_version: "encryptionkey@0"
I20260808 04:39:17.391129 10879 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260808 04:39:17.394008 10898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:17.395977 10879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:39:17.396133 10879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "aa67ef66734840c1a19f526b42be84fb"
format_stamp: "Formatted at 2026-08-08 04:39:17 on dist-test-slave-drl0"
server_key: "9546819bd5e0d78288f6b3ab125d91a2"
server_key_iv: "22e1fd2443a5d49f326fb647694ec7d9"
server_key_version: "encryptionkey@0"
I20260808 04:39:17.396261 10879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:17.418748 10879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:17.423095 10879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:17.423349 10879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:17.424194 10879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:39:17.425159 10879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:39:17.425240 10879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:17.425324 10879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:39:17.425370 10879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:17.438139 10879 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:33183
I20260808 04:39:17.438156 11048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:33183 every 8 connection(s)
I20260808 04:39:17.439553 10879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:39:17.443193 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 10879
I20260808 04:39:17.443370 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:39:17.443704 16273 external_mini_cluster.cc:1428] Setting key bf6cabb1ffcafda8a2dc99813877bb88
Aug 08 04:39:17 dist-test-slave-drl0 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163957, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.65@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.228.66@KRBTEST.COM; defaulting to no policy
I20260808 04:39:17.459285 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:17.442316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:55617 (local address 127.15.228.126:33963)
0808 04:39:17.442572 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:17.442577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:17.443635 (+ 1058us) server_negotiation.cc:407] Connection header received
0808 04:39:17.445222 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:17.445225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:17.445294 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:17.445381 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:17.446998 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:17.447817 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:17.449731 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:17.449983 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:17.450077 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:17.454006 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:17.454013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:17.454017 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:17.454069 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:17.456176 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:17.456987 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:17.456991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:17.456992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:17.457122 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:17.457638 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:17.457641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:17.457643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:17.457919 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:17.458051 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:17.458849 (+ 798us) server_negotiation.cc:299] Negotiation successful
0808 04:39:17.459114 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260808 04:39:17.460207 11051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:17.442637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:33963 (local address 127.15.228.65:55617)
0808 04:39:17.443439 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:17.443506 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:17.444986 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:17.445627 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:17.445639 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:17.446104 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:17.446667 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:17.446707 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:17.447984 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:17.447987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:17.449586 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:17.449601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:17.450161 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:17.450167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:17.450402 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:17.451406 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:17.451436 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:17.453644 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:17.456356 (+ 2712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:17.456363 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:17.456382 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:17.456826 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:17.457256 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:17.457263 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:17.457267 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:17.457486 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:17.458087 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:17.458096 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:17.458505 (+ 409us) client_negotiation.cc:769] Sending connection context
0808 04:39:17.458784 (+ 279us) client_negotiation.cc:240] Negotiation successful
0808 04:39:17.459188 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":264,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.66@KRBTEST.COM" created.
I20260808 04:39:17.461947 11049 heartbeater.cc:347] Connected to a master server at 127.15.228.126:33963
I20260808 04:39:17.462244 11049 heartbeater.cc:464] Registering TS with master...
I20260808 04:39:17.463217 11049 heartbeater.cc:511] Master 127.15.228.126:33963 requested a full tablet report, sending...
I20260808 04:39:17.465178 10788 ts_manager.cc:194] Registered new tserver with Master: aa67ef66734840c1a19f526b42be84fb (127.15.228.65:33183)
I20260808 04:39:17.466575 10788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.65@KRBTEST.COM'} at 127.15.228.65:55617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
Entry for principal kudu/127.15.228.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab.
I20260808 04:39:17.489926 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33963
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:38685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:17.624778 11055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:17.625331 11055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:17.625468 11055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:17.630697 11055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:17.630782 11055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:17.630854 11055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:39:17.637195 11055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:38685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.66.keytab
--principal=kudu/127.15.228.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:17.638545 11055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:17.639834 11055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:17.653167 11069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:17.653191 11063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:17.653167 11065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:17.653357 11055 server_base.cc:1034] running on GCE node
I20260808 04:39:17.653889 11055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:17.654627 11055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:17.655844 11055 hybrid_clock.cc:648] HybridClock initialized: now 1786163957655797 us; error 59 us; skew 500 ppm
Aug 08 04:39:17 dist-test-slave-drl0 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163957, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:17.664299 11055 init.cc:377] Logged in from keytab as kudu/127.15.228.66@KRBTEST.COM (short username kudu)
I20260808 04:39:17.665766 11055 webserver.cc:460] Webserver started at http://127.15.228.66:38637/ using document root <none> and password file <none>
I20260808 04:39:17.666466 11055 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:17.666574 11055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:17.666838 11055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:17.669370 11055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "9d8284cd6c32443fa1542a0aae6a839b"
format_stamp: "Formatted at 2026-08-08 04:39:17 on dist-test-slave-drl0"
server_key: "f247cd69c1be4a0e964c32de64094e3e"
server_key_iv: "50ac07865f84f3abe11641b4b9b8612a"
server_key_version: "encryptionkey@0"
I20260808 04:39:17.670078 11055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "9d8284cd6c32443fa1542a0aae6a839b"
format_stamp: "Formatted at 2026-08-08 04:39:17 on dist-test-slave-drl0"
server_key: "f247cd69c1be4a0e964c32de64094e3e"
server_key_iv: "50ac07865f84f3abe11641b4b9b8612a"
server_key_version: "encryptionkey@0"
I20260808 04:39:17.675069 11055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 04:39:17.678355 11076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:17.680307 11055 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:39:17.680503 11055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "9d8284cd6c32443fa1542a0aae6a839b"
format_stamp: "Formatted at 2026-08-08 04:39:17 on dist-test-slave-drl0"
server_key: "f247cd69c1be4a0e964c32de64094e3e"
server_key_iv: "50ac07865f84f3abe11641b4b9b8612a"
server_key_version: "encryptionkey@0"
I20260808 04:39:17.680751 11055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:17.701860 11055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:17.707134 11055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:17.707429 11055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:17.708395 11055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:39:17.709921 11055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:39:17.709997 11055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:17.710079 11055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:39:17.710119 11055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:17.721948 11055 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:41255
I20260808 04:39:17.722026 11233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:41255 every 8 connection(s)
I20260808 04:39:17.723070 11055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:39:17.731309 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 11055
I20260808 04:39:17.731479 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:39:17.731794 16273 external_mini_cluster.cc:1428] Setting key d86de743eb946024bc6618f44e236414
Aug 08 04:39:17 dist-test-slave-drl0 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163957, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.66@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:39:17.741974 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:17.725946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:60573 (local address 127.15.228.126:33963)
0808 04:39:17.726153 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:17.726158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:17.727737 (+ 1579us) server_negotiation.cc:407] Connection header received
0808 04:39:17.729416 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:17.729419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:17.729484 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:17.729592 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:17.731341 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:17.732036 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:17.733725 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:17.733970 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:17.734089 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:17.737005 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:17.737015 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:17.737018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:17.737042 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:17.739193 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:17.739929 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:17.739932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:17.739934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:17.740013 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:17.740472 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:17.740475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:17.740476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:17.740694 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:17.740781 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:17.741666 (+ 885us) server_negotiation.cc:299] Negotiation successful
0808 04:39:17.741821 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260808 04:39:17.743288 11237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:17.726531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:33963 (local address 127.15.228.66:60573)
0808 04:39:17.727410 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:17.727511 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:17.729103 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:17.729971 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:17.729982 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:17.730466 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:17.731154 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:17.731175 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:17.732177 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:17.732187 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:17.733562 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:17.733573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:17.734143 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:17.734147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:17.734328 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:17.735179 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:17.735196 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:17.736802 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:17.739351 (+ 2549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:17.739361 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:17.739382 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:17.739775 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:17.740114 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:17.740121 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:17.740125 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:17.740325 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:17.740813 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:17.740823 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:17.741253 (+ 430us) client_negotiation.cc:769] Sending connection context
0808 04:39:17.741510 (+ 257us) client_negotiation.cc:240] Negotiation successful
0808 04:39:17.741976 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":151,"threads_started":1}
I20260808 04:39:17.745100 11235 heartbeater.cc:347] Connected to a master server at 127.15.228.126:33963
I20260808 04:39:17.745513 11235 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.15.228.67@KRBTEST.COM; defaulting to no policy
I20260808 04:39:17.746264 11235 heartbeater.cc:511] Master 127.15.228.126:33963 requested a full tablet report, sending...
I20260808 04:39:17.747848 10788 ts_manager.cc:194] Registered new tserver with Master: 9d8284cd6c32443fa1542a0aae6a839b (127.15.228.66:41255)
I20260808 04:39:17.748687 10788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.66@KRBTEST.COM'} at 127.15.228.66:60573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
Entry for principal kudu/127.15.228.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab.
I20260808 04:39:17.771080 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33963
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:38685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:17.904696 11243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:17.905004 11243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:17.905053 11243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:17.909616 11243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:17.909682 11243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:17.909744 11243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:39:17.915208 11243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:38685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.67.keytab
--principal=kudu/127.15.228.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:33963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:17.916517 11243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:17.917723 11243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:17.928726 11250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:17.928859 11249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:17.928961 11243 server_base.cc:1034] running on GCE node
W20260808 04:39:17.928859 11252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:17.929952 11243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:17.930783 11243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:17.932008 11243 hybrid_clock.cc:648] HybridClock initialized: now 1786163957931988 us; error 34 us; skew 500 ppm
Aug 08 04:39:17 dist-test-slave-drl0 krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163957, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:17.935192 11243 init.cc:377] Logged in from keytab as kudu/127.15.228.67@KRBTEST.COM (short username kudu)
I20260808 04:39:17.936280 11243 webserver.cc:460] Webserver started at http://127.15.228.67:44063/ using document root <none> and password file <none>
I20260808 04:39:17.937000 11243 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:17.937124 11243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:17.937386 11243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:17.939584 11243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "4b3adbfdf1de4de0ba6edb212daca2f9"
format_stamp: "Formatted at 2026-08-08 04:39:17 on dist-test-slave-drl0"
server_key: "78a9b124777c7c32d5e2cd2d617b8d9a"
server_key_iv: "e2fd468d9d09020ba823ee24e790da76"
server_key_version: "encryptionkey@0"
I20260808 04:39:17.940172 11243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "4b3adbfdf1de4de0ba6edb212daca2f9"
format_stamp: "Formatted at 2026-08-08 04:39:17 on dist-test-slave-drl0"
server_key: "78a9b124777c7c32d5e2cd2d617b8d9a"
server_key_iv: "e2fd468d9d09020ba823ee24e790da76"
server_key_version: "encryptionkey@0"
I20260808 04:39:17.944327 11243 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:39:17.947573 11260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:17.949473 11243 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260808 04:39:17.949714 11243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "4b3adbfdf1de4de0ba6edb212daca2f9"
format_stamp: "Formatted at 2026-08-08 04:39:17 on dist-test-slave-drl0"
server_key: "78a9b124777c7c32d5e2cd2d617b8d9a"
server_key_iv: "e2fd468d9d09020ba823ee24e790da76"
server_key_version: "encryptionkey@0"
I20260808 04:39:17.949882 11243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:17.978499 11243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:17.983301 11243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:17.983541 11243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:17.984223 11243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:39:17.985845 11243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:39:17.985932 11243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:17.986018 11243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:39:17.986069 11243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:17.998242 11243 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:39959
I20260808 04:39:17.998258 11414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:39959 every 8 connection(s)
I20260808 04:39:17.999301 11243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:39:18.001439 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 11243
I20260808 04:39:18.001602 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:39:18.001937 16273 external_mini_cluster.cc:1428] Setting key 52839b0e5d565618ffc8e7074b51a7b0
Aug 08 04:39:18 dist-test-slave-drl0 krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163957, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.67@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:39:18.017035 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:18.004138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:39685 (local address 127.15.228.126:33963)
0808 04:39:18.004451 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:18.004459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:18.004479 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:39:18.005265 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:18.005268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:18.005325 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:18.005405 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:18.006649 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.007293 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.008633 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.008932 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.009053 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:18.012245 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:18.012248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:18.012250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:18.012275 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:18.014120 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:18.014706 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:18.014709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:18.014710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:18.014806 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:18.015306 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:18.015311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:18.015313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:18.015615 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:18.015727 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:18.016592 (+ 865us) server_negotiation.cc:299] Negotiation successful
0808 04:39:18.016855 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20260808 04:39:18.017882 11420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:18.002899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:33963 (local address 127.15.228.67:39685)
0808 04:39:18.003543 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:18.003600 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:18.005041 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:18.005592 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:18.005599 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:18.005969 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:18.006500 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:18.006513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.007432 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.007435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:18.008469 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:18.008480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.009103 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.009107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:18.009284 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:18.009986 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:18.010003 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:18.012046 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:18.014257 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:18.014262 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:18.014274 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:18.014573 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:18.014993 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:18.014997 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:18.015000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:18.015157 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:18.015740 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:18.015747 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:18.016100 (+ 353us) client_negotiation.cc:769] Sending connection context
0808 04:39:18.016302 (+ 202us) client_negotiation.cc:240] Negotiation successful
0808 04:39:18.016687 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":150,"threads_started":1}
I20260808 04:39:18.019687 11415 heartbeater.cc:347] Connected to a master server at 127.15.228.126:33963
I20260808 04:39:18.020084 11415 heartbeater.cc:464] Registering TS with master...
I20260808 04:39:18.020635 11415 heartbeater.cc:511] Master 127.15.228.126:33963 requested a full tablet report, sending...
I20260808 04:39:18.022045 10788 ts_manager.cc:194] Registered new tserver with Master: 4b3adbfdf1de4de0ba6edb212daca2f9 (127.15.228.67:39959)
I20260808 04:39:18.022842 10788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.228.67@KRBTEST.COM'} at 127.15.228.67:39685
I20260808 04:39:18.030848 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:39:18.041954 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:18.033424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38346 (local address 127.15.228.126:33963)
0808 04:39:18.033655 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:18.033660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:18.033805 (+ 145us) server_negotiation.cc:407] Connection header received
0808 04:39:18.033999 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:18.034002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:18.034048 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:18.034144 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:18.035057 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.035612 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.036827 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.037058 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.037176 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:18.038652 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:18.038656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:18.038658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:18.038689 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:18.040184 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:18.040616 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:18.040621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:18.040623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:18.040704 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:18.041014 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:18.041017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:18.041019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:18.041260 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:18.041410 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:18.041646 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 04:39:18.041778 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260808 04:39:18.045691 10788 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:39:18.051028 10788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:39:18.071134 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:18.064467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52822 (local address 127.15.228.65:33183)
0808 04:39:18.065730 (+ 1263us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:18.065740 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:18.065852 (+ 112us) server_negotiation.cc:407] Connection header received
0808 04:39:18.065973 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:18.065979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:18.066220 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:18.066780 (+ 560us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:18.066969 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.068204 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.069601 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.070388 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.070518 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:18.070690 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:18.070836 (+ 146us) server_negotiation.cc:299] Negotiation successful
0808 04:39:18.070973 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":163,"threads_started":1}
I20260808 04:39:18.071125 11431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:18.064250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:33183 (local address 127.0.0.1:52822)
0808 04:39:18.065206 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:18.065281 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:18.065675 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:18.066420 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:18.066427 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:18.066452 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:18.066622 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:18.066629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.068371 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.068374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:18.069453 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:18.069465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.070518 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.070522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:18.070630 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:18.070664 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 04:39:18.070803 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 04:39:18.070938 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":94,"thread_start_us":153,"threads_started":1}
I20260808 04:39:18.071306 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:18.064593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:41255 (local address 127.0.0.1:49822)
0808 04:39:18.065205 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:18.065276 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:18.065415 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:18.066891 (+ 1476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:18.066893 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:18.066907 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:18.067008 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:18.067013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.068350 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.068353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:18.069711 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:18.069718 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.070867 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.070871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:18.071014 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:18.071038 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 04:39:18.071095 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 04:39:18.071163 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":105,"thread_start_us":102,"threads_started":1}
I20260808 04:39:18.072290 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:18.065148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:39959 (local address 127.0.0.1:37092)
0808 04:39:18.066069 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:18.066084 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:18.066168 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:18.066782 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:18.066785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:18.066801 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:18.066992 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:18.067000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.068108 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.068112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:18.069693 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:18.069706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.071859 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.071866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:18.071973 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:18.072011 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 04:39:18.072082 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 04:39:18.072159 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"mutex_wait_us":65,"thread_start_us":110,"threads_started":1}
I20260808 04:39:18.072444 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:18.064898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.15.228.66:41255)
0808 04:39:18.066471 (+ 1573us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:18.066484 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:18.066508 (+ 24us) server_negotiation.cc:407] Connection header received
0808 04:39:18.066580 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:18.066584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:18.066756 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:18.066896 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:18.067108 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.068204 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.069846 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.070718 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.071946 (+ 1228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:18.072108 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:18.072159 (+ 51us) server_negotiation.cc:299] Negotiation successful
0808 04:39:18.072256 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1445,"thread_start_us":223,"threads_started":1}
I20260808 04:39:18.073118 11438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:18.065635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37092 (local address 127.15.228.67:39959)
0808 04:39:18.066191 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:18.066203 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:18.066226 (+ 23us) server_negotiation.cc:407] Connection header received
0808 04:39:18.066303 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:18.066307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:18.066545 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:18.067102 (+ 557us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:18.067231 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.067968 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.069846 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.070718 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.070840 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:18.071024 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:18.072697 (+ 1673us) server_negotiation.cc:299] Negotiation successful
0808 04:39:18.072816 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":182,"threads_started":1}
I20260808 04:39:18.074958 10959 tablet_service.cc:1467] Processing CreateTablet for tablet d3a4d8f19c2f48278a9d0c091c3d5602 (DEFAULT_TABLE table=test-table [id=beff3313fd0d4fa19e19fbd85dc6fe98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:39:18.075047 11145 tablet_service.cc:1467] Processing CreateTablet for tablet d3a4d8f19c2f48278a9d0c091c3d5602 (DEFAULT_TABLE table=test-table [id=beff3313fd0d4fa19e19fbd85dc6fe98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:39:18.076198 11145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a4d8f19c2f48278a9d0c091c3d5602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:18.076198 10959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a4d8f19c2f48278a9d0c091c3d5602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:18.077389 11328 tablet_service.cc:1467] Processing CreateTablet for tablet d3a4d8f19c2f48278a9d0c091c3d5602 (DEFAULT_TABLE table=test-table [id=beff3313fd0d4fa19e19fbd85dc6fe98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:39:18.078682 11328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a4d8f19c2f48278a9d0c091c3d5602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:18.086700 11440 tablet_bootstrap.cc:492] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb: Bootstrap starting.
I20260808 04:39:18.087177 11441 tablet_bootstrap.cc:492] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b: Bootstrap starting.
I20260808 04:39:18.089233 11442 tablet_bootstrap.cc:492] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9: Bootstrap starting.
I20260808 04:39:18.089380 11440 tablet_bootstrap.cc:654] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:18.089380 11441 tablet_bootstrap.cc:654] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:18.090464 11440 log.cc:824] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:18.090689 11441 log.cc:824] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:18.091490 11442 tablet_bootstrap.cc:654] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:18.092759 11442 log.cc:824] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:18.093076 11440 tablet_bootstrap.cc:492] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb: No bootstrap required, opened a new log
I20260808 04:39:18.093076 11441 tablet_bootstrap.cc:492] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b: No bootstrap required, opened a new log
I20260808 04:39:18.093299 11441 ts_tablet_manager.cc:1397] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260808 04:39:18.093299 11440 ts_tablet_manager.cc:1397] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260808 04:39:18.097947 11441 raft_consensus.cc:348] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67ef66734840c1a19f526b42be84fb" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 33183 } } peers { permanent_uuid: "4b3adbfdf1de4de0ba6edb212daca2f9" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39959 } } peers { permanent_uuid: "9d8284cd6c32443fa1542a0aae6a839b" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 41255 } }
I20260808 04:39:18.098286 11441 raft_consensus.cc:374] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:18.098340 11441 raft_consensus.cc:729] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d8284cd6c32443fa1542a0aae6a839b, State: Initialized, Role: FOLLOWER
I20260808 04:39:18.098121 11440 raft_consensus.cc:348] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67ef66734840c1a19f526b42be84fb" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 33183 } } peers { permanent_uuid: "4b3adbfdf1de4de0ba6edb212daca2f9" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39959 } } peers { permanent_uuid: "9d8284cd6c32443fa1542a0aae6a839b" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 41255 } }
I20260808 04:39:18.098404 11440 raft_consensus.cc:374] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:18.098448 11440 raft_consensus.cc:729] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa67ef66734840c1a19f526b42be84fb, State: Initialized, Role: FOLLOWER
I20260808 04:39:18.099387 11441 consensus_queue.cc:260] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67ef66734840c1a19f526b42be84fb" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 33183 } } peers { permanent_uuid: "4b3adbfdf1de4de0ba6edb212daca2f9" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39959 } } peers { permanent_uuid: "9d8284cd6c32443fa1542a0aae6a839b" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 41255 } }
I20260808 04:39:18.099387 11440 consensus_queue.cc:260] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67ef66734840c1a19f526b42be84fb" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 33183 } } peers { permanent_uuid: "4b3adbfdf1de4de0ba6edb212daca2f9" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39959 } } peers { permanent_uuid: "9d8284cd6c32443fa1542a0aae6a839b" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 41255 } }
I20260808 04:39:18.101186 11441 ts_tablet_manager.cc:1428] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260808 04:39:18.101186 11440 ts_tablet_manager.cc:1428] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260808 04:39:18.101816 11235 heartbeater.cc:503] Master 127.15.228.126:33963 was elected leader, sending a full tablet report...
I20260808 04:39:18.101866 11049 heartbeater.cc:503] Master 127.15.228.126:33963 was elected leader, sending a full tablet report...
I20260808 04:39:18.102730 11442 tablet_bootstrap.cc:492] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9: No bootstrap required, opened a new log
I20260808 04:39:18.103003 11442 ts_tablet_manager.cc:1397] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260808 04:39:18.106575 11442 raft_consensus.cc:348] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67ef66734840c1a19f526b42be84fb" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 33183 } } peers { permanent_uuid: "4b3adbfdf1de4de0ba6edb212daca2f9" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39959 } } peers { permanent_uuid: "9d8284cd6c32443fa1542a0aae6a839b" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 41255 } }
I20260808 04:39:18.106914 11442 raft_consensus.cc:374] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:18.107036 11442 raft_consensus.cc:729] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b3adbfdf1de4de0ba6edb212daca2f9, State: Initialized, Role: FOLLOWER
I20260808 04:39:18.107553 11442 consensus_queue.cc:260] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67ef66734840c1a19f526b42be84fb" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 33183 } } peers { permanent_uuid: "4b3adbfdf1de4de0ba6edb212daca2f9" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39959 } } peers { permanent_uuid: "9d8284cd6c32443fa1542a0aae6a839b" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 41255 } }
I20260808 04:39:18.108453 11415 heartbeater.cc:503] Master 127.15.228.126:33963 was elected leader, sending a full tablet report...
I20260808 04:39:18.108739 11442 ts_tablet_manager.cc:1428] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
W20260808 04:39:18.191427 11050 tablet.cc:2369] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:39:18.225083 11236 tablet.cc:2369] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:39:18.250640 11418 tablet.cc:2369] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:39:18.461149 11446 raft_consensus.cc:482] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:39:18.461459 11446 raft_consensus.cc:504] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67ef66734840c1a19f526b42be84fb" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 33183 } } peers { permanent_uuid: "4b3adbfdf1de4de0ba6edb212daca2f9" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39959 } } peers { permanent_uuid: "9d8284cd6c32443fa1542a0aae6a839b" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 41255 } }
I20260808 04:39:18.463181 11446 leader_election.cc:288] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b3adbfdf1de4de0ba6edb212daca2f9 (127.15.228.67:39959), 9d8284cd6c32443fa1542a0aae6a839b (127.15.228.66:41255)
I20260808 04:39:18.467634 11449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:18.463430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:39959 (local address 127.15.228.65:44555)
0808 04:39:18.463884 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:18.463899 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:18.464074 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:18.464377 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:18.464381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:18.464403 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:18.464532 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:18.464538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.465325 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.465329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:18.466333 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:18.466342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.467115 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.467119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:18.467226 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:18.467259 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 04:39:18.467349 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 04:39:18.467450 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":1,"thread_start_us":152,"threads_started":1}
I20260808 04:39:18.467609 11438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:18.463497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:44555 (local address 127.15.228.67:39959)
0808 04:39:18.463681 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:18.463687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:18.464001 (+ 314us) server_negotiation.cc:407] Connection header received
0808 04:39:18.464203 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:18.464206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:18.464258 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:18.464353 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:18.464650 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.465204 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.466474 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.466974 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.467115 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:18.467176 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:18.467382 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 04:39:18.467454 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 04:39:18.467983 11450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:18.463582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:41255 (local address 127.15.228.65:52673)
0808 04:39:18.464068 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:18.464081 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:18.464259 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:18.464564 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:18.464567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:18.464584 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:18.464719 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:18.464724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.465519 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.465523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:18.466477 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:18.466487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.467312 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.467316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:18.467415 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:18.467452 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 04:39:18.467697 (+ 245us) client_negotiation.cc:240] Negotiation successful
0808 04:39:18.467814 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":31,"thread_start_us":163,"threads_started":1}
I20260808 04:39:18.468348 11448 raft_consensus.cc:482] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:39:18.468458 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:18.463730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:52673 (local address 127.15.228.66:41255)
0808 04:39:18.463919 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:18.463941 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:18.464187 (+ 246us) server_negotiation.cc:407] Connection header received
0808 04:39:18.464384 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:18.464387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:18.464453 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:18.464548 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:18.464837 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.465396 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.466638 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.467177 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.467317 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:18.467386 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:18.467732 (+ 346us) server_negotiation.cc:299] Negotiation successful
0808 04:39:18.468313 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 04:39:18.468600 11448 raft_consensus.cc:504] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67ef66734840c1a19f526b42be84fb" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 33183 } } peers { permanent_uuid: "4b3adbfdf1de4de0ba6edb212daca2f9" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39959 } } peers { permanent_uuid: "9d8284cd6c32443fa1542a0aae6a839b" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 41255 } }
I20260808 04:39:18.468814 11351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a4d8f19c2f48278a9d0c091c3d5602" candidate_uuid: "aa67ef66734840c1a19f526b42be84fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b3adbfdf1de4de0ba6edb212daca2f9" is_pre_election: true
I20260808 04:39:18.469125 11351 raft_consensus.cc:2454] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa67ef66734840c1a19f526b42be84fb in term 0.
I20260808 04:39:18.469102 11170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a4d8f19c2f48278a9d0c091c3d5602" candidate_uuid: "aa67ef66734840c1a19f526b42be84fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d8284cd6c32443fa1542a0aae6a839b" is_pre_election: true
I20260808 04:39:18.469350 11170 raft_consensus.cc:2454] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa67ef66734840c1a19f526b42be84fb in term 0.
I20260808 04:39:18.469623 10901 leader_election.cc:302] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b3adbfdf1de4de0ba6edb212daca2f9, aa67ef66734840c1a19f526b42be84fb; no voters:
I20260808 04:39:18.469956 11446 raft_consensus.cc:2781] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:39:18.470057 11446 raft_consensus.cc:482] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:39:18.470111 11446 raft_consensus.cc:3037] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:18.470999 11448 leader_election.cc:288] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa67ef66734840c1a19f526b42be84fb (127.15.228.65:33183), 9d8284cd6c32443fa1542a0aae6a839b (127.15.228.66:41255)
I20260808 04:39:18.471697 11446 raft_consensus.cc:504] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67ef66734840c1a19f526b42be84fb" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 33183 } } peers { permanent_uuid: "4b3adbfdf1de4de0ba6edb212daca2f9" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39959 } } peers { permanent_uuid: "9d8284cd6c32443fa1542a0aae6a839b" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 41255 } }
I20260808 04:39:18.472065 11446 leader_election.cc:288] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb [CANDIDATE]: Term 1 election: Requested vote from peers 4b3adbfdf1de4de0ba6edb212daca2f9 (127.15.228.67:39959), 9d8284cd6c32443fa1542a0aae6a839b (127.15.228.66:41255)
I20260808 04:39:18.472946 11170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a4d8f19c2f48278a9d0c091c3d5602" candidate_uuid: "aa67ef66734840c1a19f526b42be84fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d8284cd6c32443fa1542a0aae6a839b"
I20260808 04:39:18.473080 11170 raft_consensus.cc:3037] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:18.473290 11351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a4d8f19c2f48278a9d0c091c3d5602" candidate_uuid: "aa67ef66734840c1a19f526b42be84fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b3adbfdf1de4de0ba6edb212daca2f9"
I20260808 04:39:18.473416 11351 raft_consensus.cc:3037] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:18.474610 11170 raft_consensus.cc:2454] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa67ef66734840c1a19f526b42be84fb in term 1.
I20260808 04:39:18.474754 11351 raft_consensus.cc:2454] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa67ef66734840c1a19f526b42be84fb in term 1.
I20260808 04:39:18.475147 10899 leader_election.cc:302] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d8284cd6c32443fa1542a0aae6a839b, aa67ef66734840c1a19f526b42be84fb; no voters:
I20260808 04:39:18.475458 11420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:18.470242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:33183 (local address 127.15.228.67:33017)
0808 04:39:18.470347 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:18.470363 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:18.470446 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:18.470829 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:18.470831 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:18.470845 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:18.471000 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:18.471007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.471934 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.471937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:18.473085 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:18.473098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.474988 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.474995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:18.475115 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:18.475151 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 04:39:18.475245 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 04:39:18.475332 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 04:39:18.475739 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:18.470418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:33017 (local address 127.15.228.65:33183)
0808 04:39:18.470601 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:18.470606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:18.470620 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:39:18.470658 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:18.470660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:18.470707 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:18.470791 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:18.471153 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.471806 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.473237 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.474037 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.474243 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:18.474311 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:18.475273 (+ 962us) server_negotiation.cc:299] Negotiation successful
0808 04:39:18.475410 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260808 04:39:18.475826 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:18.471043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:60797 (local address 127.15.228.66:41255)
0808 04:39:18.471168 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:18.471173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:18.471185 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:39:18.471232 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:18.471234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:18.471287 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:18.471365 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:18.471709 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.472438 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.474269 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.474876 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.475009 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:18.475064 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:18.475509 (+ 445us) server_negotiation.cc:299] Negotiation successful
0808 04:39:18.475645 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 04:39:18.475486 11446 raft_consensus.cc:2781] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:18.476208 10992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a4d8f19c2f48278a9d0c091c3d5602" candidate_uuid: "4b3adbfdf1de4de0ba6edb212daca2f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa67ef66734840c1a19f526b42be84fb" is_pre_election: true
I20260808 04:39:18.476289 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:18.470280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:41255 (local address 127.15.228.67:60797)
0808 04:39:18.470617 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:18.470628 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:18.470698 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:18.471408 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:18.471411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:18.471434 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:18.471576 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:18.471584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.472570 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.472573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:18.473980 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:18.474010 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.475207 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.475213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:18.475350 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:18.475376 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 04:39:18.476003 (+ 627us) client_negotiation.cc:240] Negotiation successful
0808 04:39:18.476130 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":125,"threads_started":1}
I20260808 04:39:18.476593 11446 raft_consensus.cc:686] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb [term 1 LEADER]: Becoming Leader. State: Replica: aa67ef66734840c1a19f526b42be84fb, State: Running, Role: LEADER
I20260808 04:39:18.476847 11170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a4d8f19c2f48278a9d0c091c3d5602" candidate_uuid: "4b3adbfdf1de4de0ba6edb212daca2f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d8284cd6c32443fa1542a0aae6a839b" is_pre_election: true
I20260808 04:39:18.477123 11170 raft_consensus.cc:2379] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4b3adbfdf1de4de0ba6edb212daca2f9 in current term 1: Already voted for candidate aa67ef66734840c1a19f526b42be84fb in this term.
I20260808 04:39:18.477299 11446 consensus_queue.cc:237] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67ef66734840c1a19f526b42be84fb" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 33183 } } peers { permanent_uuid: "4b3adbfdf1de4de0ba6edb212daca2f9" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39959 } } peers { permanent_uuid: "9d8284cd6c32443fa1542a0aae6a839b" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 41255 } }
I20260808 04:39:18.481031 11263 leader_election.cc:302] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b3adbfdf1de4de0ba6edb212daca2f9; no voters: 9d8284cd6c32443fa1542a0aae6a839b, aa67ef66734840c1a19f526b42be84fb
I20260808 04:39:18.481873 11448 raft_consensus.cc:2726] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 04:39:18.482152 10788 catalog_manager.cc:5495] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb reported cstate change: term changed from 0 to 1, leader changed from <none> to aa67ef66734840c1a19f526b42be84fb (127.15.228.65). New cstate: current_term: 1 leader_uuid: "aa67ef66734840c1a19f526b42be84fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67ef66734840c1a19f526b42be84fb" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 33183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b3adbfdf1de4de0ba6edb212daca2f9" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d8284cd6c32443fa1542a0aae6a839b" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 41255 } health_report { overall_health: UNKNOWN } } }
I20260808 04:39:18.545768 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:18.542040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52834 (local address 127.15.228.65:33183)
0808 04:39:18.542262 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:18.542266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:18.542278 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:39:18.542362 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:18.542364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:18.542421 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:18.542541 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:39:18.542822 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.543361 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.544225 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.544476 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.544589 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:18.544912 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:39:18.545360 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:39:18.545492 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:18.545555 (+ 63us) server_negotiation.cc:299] Negotiation successful
0808 04:39:18.545612 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 04:39:18.553352 11351 raft_consensus.cc:1261] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9 [term 1 FOLLOWER]: Refusing update from remote peer aa67ef66734840c1a19f526b42be84fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:39:18.553304 11170 raft_consensus.cc:1261] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b [term 1 FOLLOWER]: Refusing update from remote peer aa67ef66734840c1a19f526b42be84fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:39:18.554855 11446 consensus_queue.cc:1035] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b3adbfdf1de4de0ba6edb212daca2f9" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 39959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:39:18.555208 11452 consensus_queue.cc:1035] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d8284cd6c32443fa1542a0aae6a839b" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 41255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:39:18.566336 11458 mvcc.cc:205] Tried to move back new op lower bound from 7316127574223675392 to 7316127573927165952. Current Snapshot: MvccSnapshot[applied={T|T < 7316127574223675392}]
I20260808 04:39:18.575428 10783 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:39:18.575603 10783 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:39:18.579162 10783 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd04f6c991b44fa38839ba47725bdc10: Sending DeleteTablet for 3 replicas of tablet d3a4d8f19c2f48278a9d0c091c3d5602
I20260808 04:39:18.580122 11145 tablet_service.cc:1514] Processing DeleteTablet for tablet d3a4d8f19c2f48278a9d0c091c3d5602 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:39:18 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:49822
I20260808 04:39:18.580762 11328 tablet_service.cc:1514] Processing DeleteTablet for tablet d3a4d8f19c2f48278a9d0c091c3d5602 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:39:18 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:37092
I20260808 04:39:18.580968 10959 tablet_service.cc:1514] Processing DeleteTablet for tablet d3a4d8f19c2f48278a9d0c091c3d5602 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:39:18 UTC) from {username='kudu', principal='kudu/127.15.228.126@KRBTEST.COM'} at 127.0.0.1:52822
I20260808 04:39:18.581326 11469 tablet_replica.cc:330] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b: stopping tablet replica
I20260808 04:39:18.581408 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 10879
I20260808 04:39:18.581430 11471 tablet_replica.cc:330] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb: stopping tablet replica
I20260808 04:39:18.581789 11469 raft_consensus.cc:2229] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:39:18.581805 11471 raft_consensus.cc:2229] T d3a4d8f19c2f48278a9d0c091c3d5602 P aa67ef66734840c1a19f526b42be84fb [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:39:18.582223 11469 raft_consensus.cc:2258] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:39:18.586967 11469 ts_tablet_manager.cc:1905] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:39:18.590353 11469 ts_tablet_manager.cc:1918] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:39:18.590452 11469 log.cc:1196] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/d3a4d8f19c2f48278a9d0c091c3d5602
I20260808 04:39:18.590745 11469 ts_tablet_manager.cc:1939] T d3a4d8f19c2f48278a9d0c091c3d5602 P 9d8284cd6c32443fa1542a0aae6a839b: Deleting consensus metadata
I20260808 04:39:18.591147 11470 tablet_replica.cc:330] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9: stopping tablet replica
I20260808 04:39:18.591521 11470 raft_consensus.cc:2229] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:39:18.591856 11470 raft_consensus.cc:2258] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:39:18.600689 10767 catalog_manager.cc:4841] TS 9d8284cd6c32443fa1542a0aae6a839b (127.15.228.66:41255): tablet d3a4d8f19c2f48278a9d0c091c3d5602 (table test-table [id=beff3313fd0d4fa19e19fbd85dc6fe98]) successfully deleted
I20260808 04:39:18.602432 11470 ts_tablet_manager.cc:1905] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:39:18.606251 11470 ts_tablet_manager.cc:1918] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:39:18.606383 11470 log.cc:1196] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/d3a4d8f19c2f48278a9d0c091c3d5602
I20260808 04:39:18.607205 11470 ts_tablet_manager.cc:1939] T d3a4d8f19c2f48278a9d0c091c3d5602 P 4b3adbfdf1de4de0ba6edb212daca2f9: Deleting consensus metadata
I20260808 04:39:18.608886 10769 catalog_manager.cc:4841] TS 4b3adbfdf1de4de0ba6edb212daca2f9 (127.15.228.67:39959): tablet d3a4d8f19c2f48278a9d0c091c3d5602 (table test-table [id=beff3313fd0d4fa19e19fbd85dc6fe98]) successfully deleted
W20260808 04:39:18.609503 10769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.228.65:33183 (error 108)
I20260808 04:39:18.609880 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 11055
I20260808 04:39:18.617995 11473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:18.617283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:33183 (local address 127.0.0.1:52840)
0808 04:39:18.617701 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:18.617834 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.65:33183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":111,"threads_started":1}
W20260808 04:39:18.620504 10769 catalog_manager.cc:4584] TS aa67ef66734840c1a19f526b42be84fb (127.15.228.65:33183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d3a4d8f19c2f48278a9d0c091c3d5602: Network error: Client connection negotiation failed: client connection to 127.15.228.65:33183: connect: Connection refused (error 111)
I20260808 04:39:18.625514 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 11243
I20260808 04:39:18.639803 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 10748
2026-08-08T04:39:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (3265 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T04:39:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:18Z Disabled control of system clock
I20260808 04:39:18.684352 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:41253
--webserver_interface=127.15.228.126
--webserver_port=0
--builtin_ntp_servers=127.15.228.84:44755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:41253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:39:18.829421 11478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:18.829779 11478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:18.829823 11478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:18.834707 11478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:18.834785 11478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:18.834802 11478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:18.834820 11478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:18.834836 11478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:18.842809 11478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:44755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:41253
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:41253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:18.844352 11478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:18.846210 11478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:18.855998 11485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:18.855983 11484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:18.856068 11478 server_base.cc:1034] running on GCE node
W20260808 04:39:18.856007 11487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:18.856986 11478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:18.858673 11478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:18.859979 11478 hybrid_clock.cc:648] HybridClock initialized: now 1786163958859943 us; error 66 us; skew 500 ppm
I20260808 04:39:18.862563 11478 webserver.cc:460] Webserver started at http://127.15.228.126:39105/ using document root <none> and password file <none>
I20260808 04:39:18.863281 11478 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:18.863351 11478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:18.863531 11478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:18.865655 11478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "7ab528c6003649c8bca77b3bef102a24"
format_stamp: "Formatted at 2026-08-08 04:39:18 on dist-test-slave-drl0"
server_key: "dffbb836d6299a5f1c9506b4d271c909"
server_key_iv: "429a141b981bbabe3a4967deff02289e"
server_key_version: "encryptionkey@0"
I20260808 04:39:18.866189 11478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "7ab528c6003649c8bca77b3bef102a24"
format_stamp: "Formatted at 2026-08-08 04:39:18 on dist-test-slave-drl0"
server_key: "dffbb836d6299a5f1c9506b4d271c909"
server_key_iv: "429a141b981bbabe3a4967deff02289e"
server_key_version: "encryptionkey@0"
I20260808 04:39:18.869992 11478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260808 04:39:18.872689 11494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:18.874102 11478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:39:18.874219 11478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "7ab528c6003649c8bca77b3bef102a24"
format_stamp: "Formatted at 2026-08-08 04:39:18 on dist-test-slave-drl0"
server_key: "dffbb836d6299a5f1c9506b4d271c909"
server_key_iv: "429a141b981bbabe3a4967deff02289e"
server_key_version: "encryptionkey@0"
I20260808 04:39:18.874326 11478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:18.887655 11478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:18.888355 11478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:18.888513 11478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:18.897738 11478 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:41253
I20260808 04:39:18.897780 11574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:41253 every 8 connection(s)
I20260808 04:39:18.899241 11478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:18.903734 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 11478
I20260808 04:39:18.903927 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:18.903856 11575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:18.904260 16273 external_mini_cluster.cc:1428] Setting key f5d1921cfc03b07536bf2c9ef85be323
I20260808 04:39:18.912223 11575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24: Bootstrap starting.
I20260808 04:39:18.915632 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:18.906040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53256 (local address 127.15.228.126:41253)
0808 04:39:18.906875 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:18.906892 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:18.907084 (+ 192us) server_negotiation.cc:407] Connection header received
0808 04:39:18.907814 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:18.907834 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:18.908272 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:18.908619 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:18.909078 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.910065 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.911004 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:18.911494 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:18.911670 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:18.913008 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:18.913016 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:18.913039 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:18.913640 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:18.913728 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:18.913759 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:18.913906 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:18.914095 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 04:39:18.914517 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":201,"threads_started":1}
I20260808 04:39:18.915992 11575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:18.917362 11575 log.cc:824] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:18.920521 11575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24: No bootstrap required, opened a new log
I20260808 04:39:18.924767 11575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab528c6003649c8bca77b3bef102a24" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41253 } }
I20260808 04:39:18.925073 11575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:18.925166 11575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab528c6003649c8bca77b3bef102a24, State: Initialized, Role: FOLLOWER
I20260808 04:39:18.925696 11575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab528c6003649c8bca77b3bef102a24" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41253 } }
I20260808 04:39:18.925834 11575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:18.925879 11575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:18.925954 11575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:18.926976 11575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab528c6003649c8bca77b3bef102a24" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41253 } }
I20260808 04:39:18.927309 11575 leader_election.cc:302] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ab528c6003649c8bca77b3bef102a24; no voters:
I20260808 04:39:18.927623 11575 leader_election.cc:288] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:18.928013 11583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:18.928308 11583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24 [term 1 LEADER]: Becoming Leader. State: Replica: 7ab528c6003649c8bca77b3bef102a24, State: Running, Role: LEADER
I20260808 04:39:18.928761 11583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab528c6003649c8bca77b3bef102a24" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41253 } }
I20260808 04:39:18.929617 11575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:18.932096 11585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ab528c6003649c8bca77b3bef102a24. Latest consensus state: current_term: 1 leader_uuid: "7ab528c6003649c8bca77b3bef102a24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab528c6003649c8bca77b3bef102a24" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41253 } } }
I20260808 04:39:18.932245 11585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:18.933661 11584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ab528c6003649c8bca77b3bef102a24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab528c6003649c8bca77b3bef102a24" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41253 } } }
I20260808 04:39:18.933792 11584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:18.934564 11595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:18.938910 11595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:18.946031 11595 catalog_manager.cc:1347] Generated new cluster ID: 657687045d3e4e6f92ab23c04b2fd42c
I20260808 04:39:18.946130 11595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:18.978170 11595 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:18.979692 11595 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:18.987530 11595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24: Generated new TSK 0
I20260808 04:39:18.988799 11595 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:19.000434 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.65:0
--local_ip_for_outbound_sockets=127.15.228.65
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:41253
--builtin_ntp_servers=127.15.228.84:44755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 04:39:19.141611 11608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:19.141893 11608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:19.141935 11608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:19.147205 11608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:19.147289 11608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:19.147351 11608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.65
I20260808 04:39:19.152640 11608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:44755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.228.65
--webserver_port=0
--tserver_master_addrs=127.15.228.126:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.65
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:19.153883 11608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:19.155464 11608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:19.165817 11616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:19.165836 11618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:19.165832 11615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:19.165906 11608 server_base.cc:1034] running on GCE node
I20260808 04:39:19.166694 11608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:19.167492 11608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:19.168783 11608 hybrid_clock.cc:648] HybridClock initialized: now 1786163959168763 us; error 30 us; skew 500 ppm
I20260808 04:39:19.171154 11608 webserver.cc:460] Webserver started at http://127.15.228.65:41831/ using document root <none> and password file <none>
I20260808 04:39:19.171782 11608 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:19.171881 11608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:19.172102 11608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:19.174232 11608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-0/data/instance:
uuid: "8c42715d2526468bac083895cc466c39"
format_stamp: "Formatted at 2026-08-08 04:39:19 on dist-test-slave-drl0"
server_key: "2b7112f164f02bd5e2a30745492318b8"
server_key_iv: "96fd458c78fbf979843652d121e32973"
server_key_version: "encryptionkey@0"
I20260808 04:39:19.174829 11608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance:
uuid: "8c42715d2526468bac083895cc466c39"
format_stamp: "Formatted at 2026-08-08 04:39:19 on dist-test-slave-drl0"
server_key: "2b7112f164f02bd5e2a30745492318b8"
server_key_iv: "96fd458c78fbf979843652d121e32973"
server_key_version: "encryptionkey@0"
I20260808 04:39:19.179441 11608 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:39:19.182891 11626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:19.184859 11608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:39:19.185114 11608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-0/wal
uuid: "8c42715d2526468bac083895cc466c39"
format_stamp: "Formatted at 2026-08-08 04:39:19 on dist-test-slave-drl0"
server_key: "2b7112f164f02bd5e2a30745492318b8"
server_key_iv: "96fd458c78fbf979843652d121e32973"
server_key_version: "encryptionkey@0"
I20260808 04:39:19.185290 11608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:19.214305 11608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:19.215248 11608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:19.215494 11608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:19.216204 11608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:39:19.217301 11608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:39:19.217383 11608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:19.217463 11608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:39:19.217509 11608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:19.229166 11608 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.65:38161
I20260808 04:39:19.229188 11777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.65:38161 every 8 connection(s)
I20260808 04:39:19.230320 11608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-0/data/info.pb
I20260808 04:39:19.231106 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 11608
I20260808 04:39:19.231256 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-0/wal/instance
I20260808 04:39:19.231626 16273 external_mini_cluster.cc:1428] Setting key 015b38db4eda01ffc8892d6f63093292
I20260808 04:39:19.234789 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.66:0
--local_ip_for_outbound_sockets=127.15.228.66
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:41253
--builtin_ntp_servers=127.15.228.84:44755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:39:19.244669 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:19.232734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.65:45141 (local address 127.15.228.126:41253)
0808 04:39:19.232878 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:19.232882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:19.235088 (+ 2206us) server_negotiation.cc:407] Connection header received
0808 04:39:19.237195 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:19.237199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:19.237263 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:19.237355 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:19.239656 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.240289 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.242069 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.242517 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.242680 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:19.244063 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:19.244066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:19.244069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:19.244086 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:19.244116 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:19.244125 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:19.244207 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:19.244395 (+ 188us) server_negotiation.cc:299] Negotiation successful
0808 04:39:19.244502 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 04:39:19.245615 11782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:19.233387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:41253 (local address 127.15.228.65:45141)
0808 04:39:19.234772 (+ 1385us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:19.234835 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:19.236913 (+ 2078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:19.237835 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:19.237848 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:19.237993 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:19.239252 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:19.239284 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.240467 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.240477 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:19.241861 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:19.241877 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.242758 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.242766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:19.243017 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:19.243752 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:39:19.243753 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:19.243802 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:39:19.243879 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:19.244224 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:19.244243 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:19.244248 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 04:39:19.244382 (+ 134us) client_negotiation.cc:240] Negotiation successful
0808 04:39:19.244712 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":143,"threads_started":1}
I20260808 04:39:19.246981 11778 heartbeater.cc:347] Connected to a master server at 127.15.228.126:41253
I20260808 04:39:19.247267 11778 heartbeater.cc:464] Registering TS with master...
I20260808 04:39:19.247871 11778 heartbeater.cc:511] Master 127.15.228.126:41253 requested a full tablet report, sending...
I20260808 04:39:19.249567 11517 ts_manager.cc:194] Registered new tserver with Master: 8c42715d2526468bac083895cc466c39 (127.15.228.65:38161)
I20260808 04:39:19.250953 11517 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.228.65:45141
W20260808 04:39:19.367753 11784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:19.368079 11784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:19.368147 11784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:19.373271 11784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:19.373376 11784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:19.373469 11784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.66
I20260808 04:39:19.378852 11784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:44755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.228.66
--webserver_port=0
--tserver_master_addrs=127.15.228.126:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.66
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:19.380223 11784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:19.381762 11784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:19.393785 11795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:19.393787 11791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:19.394054 11793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:19.394392 11784 server_base.cc:1034] running on GCE node
I20260808 04:39:19.394969 11784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:19.395720 11784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:19.396938 11784 hybrid_clock.cc:648] HybridClock initialized: now 1786163959396918 us; error 37 us; skew 500 ppm
I20260808 04:39:19.399341 11784 webserver.cc:460] Webserver started at http://127.15.228.66:38471/ using document root <none> and password file <none>
I20260808 04:39:19.399981 11784 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:19.400050 11784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:19.400278 11784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:19.402549 11784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-1/data/instance:
uuid: "45a9f7cf20ff4ccd8a8bd8464b55a185"
format_stamp: "Formatted at 2026-08-08 04:39:19 on dist-test-slave-drl0"
server_key: "3c284e0c5914acfa4415a100dc57c4e8"
server_key_iv: "cc77c46041c31db735ba5fb8ee4b9b88"
server_key_version: "encryptionkey@0"
I20260808 04:39:19.403929 11784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance:
uuid: "45a9f7cf20ff4ccd8a8bd8464b55a185"
format_stamp: "Formatted at 2026-08-08 04:39:19 on dist-test-slave-drl0"
server_key: "3c284e0c5914acfa4415a100dc57c4e8"
server_key_iv: "cc77c46041c31db735ba5fb8ee4b9b88"
server_key_version: "encryptionkey@0"
I20260808 04:39:19.409675 11784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:39:19.413573 11803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:19.415474 11784 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:39:19.415652 11784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-1/wal
uuid: "45a9f7cf20ff4ccd8a8bd8464b55a185"
format_stamp: "Formatted at 2026-08-08 04:39:19 on dist-test-slave-drl0"
server_key: "3c284e0c5914acfa4415a100dc57c4e8"
server_key_iv: "cc77c46041c31db735ba5fb8ee4b9b88"
server_key_version: "encryptionkey@0"
I20260808 04:39:19.415804 11784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:19.440054 11784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:19.440853 11784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:19.441090 11784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:19.442404 11784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:39:19.444011 11784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:39:19.444118 11784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:19.444201 11784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:39:19.444247 11784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:19.458737 11784 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.66:39773
I20260808 04:39:19.458822 11971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.66:39773 every 8 connection(s)
I20260808 04:39:19.459826 11784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-1/data/info.pb
I20260808 04:39:19.465526 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 11784
I20260808 04:39:19.465718 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-1/wal/instance
I20260808 04:39:19.466006 16273 external_mini_cluster.cc:1428] Setting key 16026426733e86d06e3f8b2af67deec2
I20260808 04:39:19.468930 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.228.67:0
--local_ip_for_outbound_sockets=127.15.228.67
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:41253
--builtin_ntp_servers=127.15.228.84:44755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 04:39:19.470656 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:19.462015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.66:48305 (local address 127.15.228.126:41253)
0808 04:39:19.462167 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:19.462171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:19.463566 (+ 1395us) server_negotiation.cc:407] Connection header received
0808 04:39:19.465065 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:19.465068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:19.465130 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:19.465247 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:19.466312 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.466996 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.468499 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.468726 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.468858 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:19.470122 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:19.470126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:19.470129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:19.470150 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:19.470169 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:19.470178 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:19.470261 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:19.470434 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 04:39:19.470501 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:39:19.472041 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:19.462204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:41253 (local address 127.15.228.66:48305)
0808 04:39:19.463417 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:19.463462 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:19.464839 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:19.465445 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:19.465452 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:19.465537 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:19.466167 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:19.466182 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.467139 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.467143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:19.468365 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:19.468375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.468906 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.468911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:19.469083 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:19.469838 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:39:19.469840 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:19.469885 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:39:19.469956 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:19.470268 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:19.470280 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:19.470286 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 04:39:19.470434 (+ 148us) client_negotiation.cc:240] Negotiation successful
0808 04:39:19.470742 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":450,"threads_started":1}
I20260808 04:39:19.473495 11972 heartbeater.cc:347] Connected to a master server at 127.15.228.126:41253
I20260808 04:39:19.473778 11972 heartbeater.cc:464] Registering TS with master...
I20260808 04:39:19.474387 11972 heartbeater.cc:511] Master 127.15.228.126:41253 requested a full tablet report, sending...
I20260808 04:39:19.475980 11517 ts_manager.cc:194] Registered new tserver with Master: 45a9f7cf20ff4ccd8a8bd8464b55a185 (127.15.228.66:39773)
I20260808 04:39:19.476953 11517 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.228.66:48305
W20260808 04:39:19.595995 11977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:19.596326 11977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:19.596414 11977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:19.600747 11977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:19.600870 11977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:19.600986 11977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.228.67
I20260808 04:39:19.606874 11977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:44755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.228.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.228.67
--webserver_port=0
--tserver_master_addrs=127.15.228.126:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.228.67
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:19.608834 11977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:19.610810 11977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:39:19.623790 11977 server_base.cc:1034] running on GCE node
W20260808 04:39:19.623708 11983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:19.623700 11982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:19.623723 11985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:19.624629 11977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:19.625656 11977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:19.627151 11977 hybrid_clock.cc:648] HybridClock initialized: now 1786163959627107 us; error 66 us; skew 500 ppm
I20260808 04:39:19.630079 11977 webserver.cc:460] Webserver started at http://127.15.228.67:42073/ using document root <none> and password file <none>
I20260808 04:39:19.630998 11977 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:19.631088 11977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:19.631433 11977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:19.634528 11977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-2/data/instance:
uuid: "6cca44f118ee4502ab69d73680cd319e"
format_stamp: "Formatted at 2026-08-08 04:39:19 on dist-test-slave-drl0"
server_key: "cd9c0fe247b0600748a0840e22783fea"
server_key_iv: "ec7616ed5cec6a7e5e3a94c23e16e307"
server_key_version: "encryptionkey@0"
I20260808 04:39:19.635682 11977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance:
uuid: "6cca44f118ee4502ab69d73680cd319e"
format_stamp: "Formatted at 2026-08-08 04:39:19 on dist-test-slave-drl0"
server_key: "cd9c0fe247b0600748a0840e22783fea"
server_key_iv: "ec7616ed5cec6a7e5e3a94c23e16e307"
server_key_version: "encryptionkey@0"
I20260808 04:39:19.641130 11977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:39:19.644673 11992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:19.646562 11977 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:39:19.646883 11977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-2/wal
uuid: "6cca44f118ee4502ab69d73680cd319e"
format_stamp: "Formatted at 2026-08-08 04:39:19 on dist-test-slave-drl0"
server_key: "cd9c0fe247b0600748a0840e22783fea"
server_key_iv: "ec7616ed5cec6a7e5e3a94c23e16e307"
server_key_version: "encryptionkey@0"
I20260808 04:39:19.647221 11977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:19.691020 11977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:19.691886 11977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:19.692129 11977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:19.692855 11977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 04:39:19.693994 11977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 04:39:19.694083 11977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:19.694166 11977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 04:39:19.694216 11977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:19.706578 11977 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.67:45517
I20260808 04:39:19.706629 12149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.67:45517 every 8 connection(s)
I20260808 04:39:19.707970 11977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-2/data/info.pb
I20260808 04:39:19.709784 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 11977
I20260808 04:39:19.709946 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-2/wal/instance
I20260808 04:39:19.710316 16273 external_mini_cluster.cc:1428] Setting key e7b625c86d9a4a2d628aae24085215c0
I20260808 04:39:19.722074 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:19.710655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:56903 (local address 127.15.228.126:41253)
0808 04:39:19.710824 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:19.710831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:19.712657 (+ 1826us) server_negotiation.cc:407] Connection header received
0808 04:39:19.715180 (+ 2523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:19.715183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:19.715244 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:19.715333 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:19.716678 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.717430 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.719315 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.719608 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.719751 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:19.721392 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:19.721396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:19.721398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:19.721415 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:19.721434 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:19.721442 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:19.721506 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:19.721886 (+ 380us) server_negotiation.cc:299] Negotiation successful
0808 04:39:19.721950 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 04:39:19.723639 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:19.711276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:41253 (local address 127.15.228.67:56903)
0808 04:39:19.712453 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:19.712516 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:19.714418 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:19.715557 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:19.715566 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:19.715690 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:19.716426 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:19.716443 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.717623 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.717627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:19.719091 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:19.719112 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.719826 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.719833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:19.720037 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:19.721059 (+ 1022us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 04:39:19.721061 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:19.721108 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 04:39:19.721184 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:19.721606 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:19.721621 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:19.721630 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 04:39:19.721821 (+ 191us) client_negotiation.cc:240] Negotiation successful
0808 04:39:19.722342 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":186,"threads_started":1}
I20260808 04:39:19.725104 12152 heartbeater.cc:347] Connected to a master server at 127.15.228.126:41253
I20260808 04:39:19.725411 12152 heartbeater.cc:464] Registering TS with master...
I20260808 04:39:19.726042 12152 heartbeater.cc:511] Master 127.15.228.126:41253 requested a full tablet report, sending...
I20260808 04:39:19.727699 11516 ts_manager.cc:194] Registered new tserver with Master: 6cca44f118ee4502ab69d73680cd319e (127.15.228.67:45517)
I20260808 04:39:19.728340 11516 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.228.67:56903
I20260808 04:39:19.738095 16273 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 04:39:19.745529 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:19.740379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53258 (local address 127.15.228.126:41253)
0808 04:39:19.740536 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:19.740540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:19.740919 (+ 379us) server_negotiation.cc:407] Connection header received
0808 04:39:19.740986 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:19.740988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:19.741035 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:19.741105 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:19.741497 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.742040 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.743323 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.743544 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.743645 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:19.744753 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:19.744756 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:19.744758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:19.744770 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:19.744791 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:19.744799 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:19.744887 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:19.745118 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 04:39:19.745412 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 04:39:19.749739 11516 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 04:39:19.753578 11516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 04:39:19.770612 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:19.764511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:38161 (local address 127.0.0.1:54450)
0808 04:39:19.765244 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:19.765295 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:19.765467 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:19.766198 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:19.766203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:19.766238 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:19.766441 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:19.766451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.767686 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.767692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:19.769227 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:19.769238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.770032 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.770035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:19.770138 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:19.770175 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 04:39:19.770301 (+ 126us) client_negotiation.cc:240] Negotiation successful
0808 04:39:19.770444 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":61,"spinlock_wait_cycles":3968,"thread_start_us":91,"threads_started":1}
I20260808 04:39:19.770681 12169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:19.764643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.15.228.65:38161)
0808 04:39:19.764986 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:19.764996 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:19.765394 (+ 398us) server_negotiation.cc:407] Connection header received
0808 04:39:19.765640 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:19.765646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:19.765851 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:19.765982 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:19.766588 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.767583 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.769356 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.769888 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.770024 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:19.770231 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:19.770382 (+ 151us) server_negotiation.cc:299] Negotiation successful
0808 04:39:19.770505 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":132,"threads_started":1}
I20260808 04:39:19.771186 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:19.765341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.67:45517 (local address 127.0.0.1:44036)
0808 04:39:19.765752 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:19.765766 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:19.765868 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:19.767097 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:19.767100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:19.767120 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:19.767204 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:19.767210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.768364 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.768370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:19.769546 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:19.769556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.770779 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.770784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:19.770886 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:19.770917 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 04:39:19.771027 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 04:39:19.771082 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":145,"thread_start_us":95,"threads_started":1}
I20260808 04:39:19.772181 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:19.766046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44036 (local address 127.15.228.67:45517)
0808 04:39:19.766619 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:19.766633 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:19.766654 (+ 21us) server_negotiation.cc:407] Connection header received
0808 04:39:19.766731 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:19.766748 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:19.766919 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:19.767102 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:19.767336 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.768253 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.769674 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.770609 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.771410 (+ 801us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:19.771579 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:19.771622 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 04:39:19.771718 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":107,"threads_started":1}
I20260808 04:39:19.772469 12168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:19.764833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:39773 (local address 127.0.0.1:43760)
0808 04:39:19.765305 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:19.765319 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:19.765447 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:19.766203 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:19.766206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:19.766247 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:19.766474 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:19.766484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.767933 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.767937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:19.770339 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:19.770347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.771770 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.771774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:19.771928 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:19.771963 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 04:39:19.772057 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 04:39:19.772131 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":1274,"spinlock_wait_cycles":4736,"thread_start_us":124,"threads_started":1}
I20260808 04:39:19.773123 12176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:19.765334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43760 (local address 127.15.228.66:39773)
0808 04:39:19.765752 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:19.765765 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:19.765792 (+ 27us) server_negotiation.cc:407] Connection header received
0808 04:39:19.765875 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:19.765880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:19.766058 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:19.766220 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:19.766619 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.767803 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.770471 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:19.771629 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:19.771770 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:19.771944 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:19.772081 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 04:39:19.772234 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":109,"threads_started":1}
I20260808 04:39:19.774694 11695 tablet_service.cc:1467] Processing CreateTablet for tablet a1889d3d790445af8f83ee6575b7f875 (DEFAULT_TABLE table=test-table [id=3c61a03284d04f22b9e64fc7ffb6efdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:39:19.774834 12059 tablet_service.cc:1467] Processing CreateTablet for tablet a1889d3d790445af8f83ee6575b7f875 (DEFAULT_TABLE table=test-table [id=3c61a03284d04f22b9e64fc7ffb6efdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:39:19.776055 11695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1889d3d790445af8f83ee6575b7f875. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:19.776145 12059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1889d3d790445af8f83ee6575b7f875. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:19.775885 11871 tablet_service.cc:1467] Processing CreateTablet for tablet a1889d3d790445af8f83ee6575b7f875 (DEFAULT_TABLE table=test-table [id=3c61a03284d04f22b9e64fc7ffb6efdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 04:39:19.776795 11871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1889d3d790445af8f83ee6575b7f875. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:19.783067 12179 tablet_bootstrap.cc:492] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e: Bootstrap starting.
I20260808 04:39:19.784068 12181 tablet_bootstrap.cc:492] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185: Bootstrap starting.
I20260808 04:39:19.784577 12180 tablet_bootstrap.cc:492] T a1889d3d790445af8f83ee6575b7f875 P 8c42715d2526468bac083895cc466c39: Bootstrap starting.
I20260808 04:39:19.785887 12179 tablet_bootstrap.cc:654] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:19.786391 12181 tablet_bootstrap.cc:654] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:19.786679 12180 tablet_bootstrap.cc:654] T a1889d3d790445af8f83ee6575b7f875 P 8c42715d2526468bac083895cc466c39: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:19.786711 12179 log.cc:824] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:19.787240 12181 log.cc:824] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:19.787475 12180 log.cc:824] T a1889d3d790445af8f83ee6575b7f875 P 8c42715d2526468bac083895cc466c39: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:19.788890 12179 tablet_bootstrap.cc:492] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e: No bootstrap required, opened a new log
I20260808 04:39:19.789104 12179 ts_tablet_manager.cc:1397] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 04:39:19.789495 12180 tablet_bootstrap.cc:492] T a1889d3d790445af8f83ee6575b7f875 P 8c42715d2526468bac083895cc466c39: No bootstrap required, opened a new log
I20260808 04:39:19.789674 12180 ts_tablet_manager.cc:1397] T a1889d3d790445af8f83ee6575b7f875 P 8c42715d2526468bac083895cc466c39: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 04:39:19.790050 12181 tablet_bootstrap.cc:492] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185: No bootstrap required, opened a new log
I20260808 04:39:19.790230 12181 ts_tablet_manager.cc:1397] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 04:39:19.792877 12179 raft_consensus.cc:348] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c42715d2526468bac083895cc466c39" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38161 } } peers { permanent_uuid: "45a9f7cf20ff4ccd8a8bd8464b55a185" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39773 } } peers { permanent_uuid: "6cca44f118ee4502ab69d73680cd319e" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 45517 } }
I20260808 04:39:19.793123 12179 raft_consensus.cc:374] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:19.793185 12179 raft_consensus.cc:729] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cca44f118ee4502ab69d73680cd319e, State: Initialized, Role: FOLLOWER
I20260808 04:39:19.793615 12179 consensus_queue.cc:260] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c42715d2526468bac083895cc466c39" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38161 } } peers { permanent_uuid: "45a9f7cf20ff4ccd8a8bd8464b55a185" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39773 } } peers { permanent_uuid: "6cca44f118ee4502ab69d73680cd319e" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 45517 } }
I20260808 04:39:19.794273 12152 heartbeater.cc:503] Master 127.15.228.126:41253 was elected leader, sending a full tablet report...
I20260808 04:39:19.795539 12181 raft_consensus.cc:348] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c42715d2526468bac083895cc466c39" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38161 } } peers { permanent_uuid: "45a9f7cf20ff4ccd8a8bd8464b55a185" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39773 } } peers { permanent_uuid: "6cca44f118ee4502ab69d73680cd319e" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 45517 } }
I20260808 04:39:19.795491 12180 raft_consensus.cc:348] T a1889d3d790445af8f83ee6575b7f875 P 8c42715d2526468bac083895cc466c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c42715d2526468bac083895cc466c39" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38161 } } peers { permanent_uuid: "45a9f7cf20ff4ccd8a8bd8464b55a185" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39773 } } peers { permanent_uuid: "6cca44f118ee4502ab69d73680cd319e" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 45517 } }
I20260808 04:39:19.795775 12180 raft_consensus.cc:374] T a1889d3d790445af8f83ee6575b7f875 P 8c42715d2526468bac083895cc466c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:19.795775 12181 raft_consensus.cc:374] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:19.795838 12180 raft_consensus.cc:729] T a1889d3d790445af8f83ee6575b7f875 P 8c42715d2526468bac083895cc466c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c42715d2526468bac083895cc466c39, State: Initialized, Role: FOLLOWER
I20260808 04:39:19.795852 12181 raft_consensus.cc:729] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a9f7cf20ff4ccd8a8bd8464b55a185, State: Initialized, Role: FOLLOWER
I20260808 04:39:19.796038 12179 ts_tablet_manager.cc:1428] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260808 04:39:19.796345 12180 consensus_queue.cc:260] T a1889d3d790445af8f83ee6575b7f875 P 8c42715d2526468bac083895cc466c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c42715d2526468bac083895cc466c39" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38161 } } peers { permanent_uuid: "45a9f7cf20ff4ccd8a8bd8464b55a185" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39773 } } peers { permanent_uuid: "6cca44f118ee4502ab69d73680cd319e" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 45517 } }
I20260808 04:39:19.796345 12181 consensus_queue.cc:260] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c42715d2526468bac083895cc466c39" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38161 } } peers { permanent_uuid: "45a9f7cf20ff4ccd8a8bd8464b55a185" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39773 } } peers { permanent_uuid: "6cca44f118ee4502ab69d73680cd319e" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 45517 } }
I20260808 04:39:19.797158 11778 heartbeater.cc:503] Master 127.15.228.126:41253 was elected leader, sending a full tablet report...
I20260808 04:39:19.797363 12181 ts_tablet_manager.cc:1428] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260808 04:39:19.797376 12180 ts_tablet_manager.cc:1428] T a1889d3d790445af8f83ee6575b7f875 P 8c42715d2526468bac083895cc466c39: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260808 04:39:19.798166 11972 heartbeater.cc:503] Master 127.15.228.126:41253 was elected leader, sending a full tablet report...
W20260808 04:39:19.959358 12153 tablet.cc:2369] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:39:19.961301 11973 tablet.cc:2369] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 04:39:19.982086 11781 tablet.cc:2369] T a1889d3d790445af8f83ee6575b7f875 P 8c42715d2526468bac083895cc466c39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:39:20.006865 12188 raft_consensus.cc:482] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 04:39:20.007254 12188 raft_consensus.cc:504] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c42715d2526468bac083895cc466c39" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38161 } } peers { permanent_uuid: "45a9f7cf20ff4ccd8a8bd8464b55a185" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39773 } } peers { permanent_uuid: "6cca44f118ee4502ab69d73680cd319e" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 45517 } }
I20260808 04:39:20.008682 12188 leader_election.cc:288] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c42715d2526468bac083895cc466c39 (127.15.228.65:38161), 45a9f7cf20ff4ccd8a8bd8464b55a185 (127.15.228.66:39773)
I20260808 04:39:20.012625 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:20.008882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:38161 (local address 127.15.228.67:58951)
0808 04:39:20.009018 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:20.009037 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:20.009139 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:20.009612 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:20.009615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:20.009634 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:20.009778 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:20.009785 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:20.010544 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:20.010549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:20.011492 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:20.011503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:20.012151 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:20.012154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:20.012273 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:20.012304 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 04:39:20.012397 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 04:39:20.012489 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":3}
I20260808 04:39:20.012625 12169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:20.009145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:58951 (local address 127.15.228.65:38161)
0808 04:39:20.009381 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:20.009388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:20.009408 (+ 20us) server_negotiation.cc:407] Connection header received
0808 04:39:20.009458 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:20.009461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:20.009500 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:20.009593 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:20.009903 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:20.010424 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:20.011626 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:20.012026 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:20.012160 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:20.012229 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:20.012414 (+ 185us) server_negotiation.cc:299] Negotiation successful
0808 04:39:20.012487 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260808 04:39:20.013509 11722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1889d3d790445af8f83ee6575b7f875" candidate_uuid: "6cca44f118ee4502ab69d73680cd319e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c42715d2526468bac083895cc466c39" is_pre_election: true
I20260808 04:39:20.014122 11722 raft_consensus.cc:2454] T a1889d3d790445af8f83ee6575b7f875 P 8c42715d2526468bac083895cc466c39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cca44f118ee4502ab69d73680cd319e in term 0.
I20260808 04:39:20.014187 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:20.009161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.66:39773 (local address 127.15.228.67:56373)
0808 04:39:20.009862 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:20.009874 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:20.010007 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:20.010481 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:20.010484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:20.010511 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 04:39:20.010731 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:20.010738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:20.011623 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:20.011629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:20.012487 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:20.012496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:20.013710 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:20.013714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:20.013837 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:20.013856 (+ 19us) client_negotiation.cc:769] Sending connection context
0808 04:39:20.013965 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 04:39:20.014047 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":36,"thread_start_us":344,"threads_started":1}
I20260808 04:39:20.014187 12176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:20.009586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.228.67:56373 (local address 127.15.228.66:39773)
0808 04:39:20.009749 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:20.009754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:20.009968 (+ 214us) server_negotiation.cc:407] Connection header received
0808 04:39:20.010268 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:20.010272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:20.010353 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:20.010460 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 04:39:20.010874 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:20.011473 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:20.012633 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:20.013529 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:20.013704 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:20.013756 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:20.013986 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 04:39:20.014076 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 04:39:20.014833 11900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1889d3d790445af8f83ee6575b7f875" candidate_uuid: "6cca44f118ee4502ab69d73680cd319e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a9f7cf20ff4ccd8a8bd8464b55a185" is_pre_election: true
I20260808 04:39:20.014894 11993 leader_election.cc:302] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cca44f118ee4502ab69d73680cd319e, 8c42715d2526468bac083895cc466c39; no voters:
I20260808 04:39:20.015206 11900 raft_consensus.cc:2454] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cca44f118ee4502ab69d73680cd319e in term 0.
I20260808 04:39:20.015250 12188 raft_consensus.cc:2781] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 04:39:20.015372 12188 raft_consensus.cc:482] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 04:39:20.015451 12188 raft_consensus.cc:3037] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:20.017081 12188 raft_consensus.cc:504] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c42715d2526468bac083895cc466c39" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38161 } } peers { permanent_uuid: "45a9f7cf20ff4ccd8a8bd8464b55a185" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39773 } } peers { permanent_uuid: "6cca44f118ee4502ab69d73680cd319e" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 45517 } }
I20260808 04:39:20.017501 12188 leader_election.cc:288] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [CANDIDATE]: Term 1 election: Requested vote from peers 8c42715d2526468bac083895cc466c39 (127.15.228.65:38161), 45a9f7cf20ff4ccd8a8bd8464b55a185 (127.15.228.66:39773)
I20260808 04:39:20.017920 11900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1889d3d790445af8f83ee6575b7f875" candidate_uuid: "6cca44f118ee4502ab69d73680cd319e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a9f7cf20ff4ccd8a8bd8464b55a185"
I20260808 04:39:20.018077 11900 raft_consensus.cc:3037] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:20.018330 11722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1889d3d790445af8f83ee6575b7f875" candidate_uuid: "6cca44f118ee4502ab69d73680cd319e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c42715d2526468bac083895cc466c39"
I20260808 04:39:20.018586 11722 raft_consensus.cc:3037] T a1889d3d790445af8f83ee6575b7f875 P 8c42715d2526468bac083895cc466c39 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:20.019807 11900 raft_consensus.cc:2454] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cca44f118ee4502ab69d73680cd319e in term 1.
I20260808 04:39:20.019969 11722 raft_consensus.cc:2454] T a1889d3d790445af8f83ee6575b7f875 P 8c42715d2526468bac083895cc466c39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cca44f118ee4502ab69d73680cd319e in term 1.
I20260808 04:39:20.020181 11996 leader_election.cc:302] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a9f7cf20ff4ccd8a8bd8464b55a185, 6cca44f118ee4502ab69d73680cd319e; no voters:
I20260808 04:39:20.020408 12188 raft_consensus.cc:2781] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:20.020645 12188 raft_consensus.cc:686] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [term 1 LEADER]: Becoming Leader. State: Replica: 6cca44f118ee4502ab69d73680cd319e, State: Running, Role: LEADER
I20260808 04:39:20.021442 12188 consensus_queue.cc:237] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c42715d2526468bac083895cc466c39" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38161 } } peers { permanent_uuid: "45a9f7cf20ff4ccd8a8bd8464b55a185" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39773 } } peers { permanent_uuid: "6cca44f118ee4502ab69d73680cd319e" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 45517 } }
I20260808 04:39:20.026302 11516 catalog_manager.cc:5495] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cca44f118ee4502ab69d73680cd319e (127.15.228.67). New cstate: current_term: 1 leader_uuid: "6cca44f118ee4502ab69d73680cd319e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c42715d2526468bac083895cc466c39" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45a9f7cf20ff4ccd8a8bd8464b55a185" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cca44f118ee4502ab69d73680cd319e" member_type: VOTER last_known_addr { host: "127.15.228.67" port: 45517 } health_report { overall_health: HEALTHY } } }
I20260808 04:39:20.077013 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:20.073079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44048 (local address 127.15.228.67:45517)
0808 04:39:20.073244 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:20.073249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:20.073262 (+ 13us) server_negotiation.cc:407] Connection header received
0808 04:39:20.073371 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:20.073374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:20.073433 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:20.073548 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:39:20.073810 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:20.074342 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:20.075307 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:20.075526 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:20.075776 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:20.076119 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:39:20.076578 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:39:20.076731 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:20.076796 (+ 65us) server_negotiation.cc:299] Negotiation successful
0808 04:39:20.076850 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:39:20.084563 11900 raft_consensus.cc:1261] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185 [term 1 FOLLOWER]: Refusing update from remote peer 6cca44f118ee4502ab69d73680cd319e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:39:20.085078 11722 raft_consensus.cc:1261] T a1889d3d790445af8f83ee6575b7f875 P 8c42715d2526468bac083895cc466c39 [term 1 FOLLOWER]: Refusing update from remote peer 6cca44f118ee4502ab69d73680cd319e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:39:20.085723 12192 consensus_queue.cc:1035] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a9f7cf20ff4ccd8a8bd8464b55a185" member_type: VOTER last_known_addr { host: "127.15.228.66" port: 39773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:39:20.086148 12188 consensus_queue.cc:1035] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c42715d2526468bac083895cc466c39" member_type: VOTER last_known_addr { host: "127.15.228.65" port: 38161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:39:20.097210 12198 mvcc.cc:205] Tried to move back new op lower bound from 7316127580496969728 to 7316127580252102656. Current Snapshot: MvccSnapshot[applied={T|T < 7316127580496969728 or (T in {7316127580496969728})}]
I20260808 04:39:20.104372 11517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:39:20.104570 11517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 04:39:20.107092 11517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ab528c6003649c8bca77b3bef102a24: Sending DeleteTablet for 3 replicas of tablet a1889d3d790445af8f83ee6575b7f875
I20260808 04:39:20.108060 12059 tablet_service.cc:1514] Processing DeleteTablet for tablet a1889d3d790445af8f83ee6575b7f875 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:39:20 UTC) from {username='slave'} at 127.0.0.1:44036
I20260808 04:39:20.108130 11695 tablet_service.cc:1514] Processing DeleteTablet for tablet a1889d3d790445af8f83ee6575b7f875 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:39:20 UTC) from {username='slave'} at 127.0.0.1:54450
I20260808 04:39:20.108261 11871 tablet_service.cc:1514] Processing DeleteTablet for tablet a1889d3d790445af8f83ee6575b7f875 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 04:39:20 UTC) from {username='slave'} at 127.0.0.1:43760
I20260808 04:39:20.108615 12214 tablet_replica.cc:330] T a1889d3d790445af8f83ee6575b7f875 P 8c42715d2526468bac083895cc466c39: stopping tablet replica
I20260808 04:39:20.108845 12215 tablet_replica.cc:330] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185: stopping tablet replica
I20260808 04:39:20.108994 12214 raft_consensus.cc:2229] T a1889d3d790445af8f83ee6575b7f875 P 8c42715d2526468bac083895cc466c39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:39:20.109251 12215 raft_consensus.cc:2229] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 04:39:20.109366 12214 raft_consensus.cc:2258] T a1889d3d790445af8f83ee6575b7f875 P 8c42715d2526468bac083895cc466c39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:39:20.109562 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 11608
I20260808 04:39:20.113039 12212 tablet_replica.cc:330] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e: stopping tablet replica
I20260808 04:39:20.113349 12212 raft_consensus.cc:2229] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [term 1 LEADER]: Raft consensus shutting down.
I20260808 04:39:20.113701 12212 raft_consensus.cc:2258] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:39:20.115182 12215 raft_consensus.cc:2258] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 04:39:20.115861 12212 ts_tablet_manager.cc:1905] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:39:20.116914 12215 ts_tablet_manager.cc:1905] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 04:39:20.119246 12212 ts_tablet_manager.cc:1918] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:39:20.119362 12212 log.cc:1196] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-2/wal/wals/a1889d3d790445af8f83ee6575b7f875
I20260808 04:39:20.119777 12212 ts_tablet_manager.cc:1939] T a1889d3d790445af8f83ee6575b7f875 P 6cca44f118ee4502ab69d73680cd319e: Deleting consensus metadata
I20260808 04:39:20.120966 12215 ts_tablet_manager.cc:1918] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 04:39:20.121273 12215 log.cc:1196] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185: Deleting WAL directory at /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786163668922094-16273-0/minicluster-data/ts-1/wal/wals/a1889d3d790445af8f83ee6575b7f875
I20260808 04:39:20.121816 12215 ts_tablet_manager.cc:1939] T a1889d3d790445af8f83ee6575b7f875 P 45a9f7cf20ff4ccd8a8bd8464b55a185: Deleting consensus metadata
I20260808 04:39:20.121950 11496 catalog_manager.cc:4841] TS 6cca44f118ee4502ab69d73680cd319e (127.15.228.67:45517): tablet a1889d3d790445af8f83ee6575b7f875 (table test-table [id=3c61a03284d04f22b9e64fc7ffb6efdb]) successfully deleted
I20260808 04:39:20.123579 11498 catalog_manager.cc:4841] TS 45a9f7cf20ff4ccd8a8bd8464b55a185 (127.15.228.66:39773): tablet a1889d3d790445af8f83ee6575b7f875 (table test-table [id=3c61a03284d04f22b9e64fc7ffb6efdb]) successfully deleted
W20260808 04:39:20.124919 11496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.228.65:38161 (error 108)
I20260808 04:39:20.125197 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 11784
I20260808 04:39:20.132266 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:20.131778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.65:38161 (local address 127.0.0.1:54460)
0808 04:39:20.132019 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:20.132135 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.65:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260808 04:39:20.132514 11496 catalog_manager.cc:4584] TS 8c42715d2526468bac083895cc466c39 (127.15.228.65:38161): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1889d3d790445af8f83ee6575b7f875: Network error: Client connection negotiation failed: client connection to 127.15.228.65:38161: connect: Connection refused (error 111)
I20260808 04:39:20.141559 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 11977
I20260808 04:39:20.157325 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 11478
2026-08-08T04:39:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1515 ms)
[----------] 6 tests from EncryptionPolicyTest (13232 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:20 dist-test-slave-drl0 krb5kdc[12221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:20 dist-test-slave-drl0 krb5kdc[12221](info): set up 2 sockets
Aug 08 04:39:20 dist-test-slave-drl0 krb5kdc[12221](info): commencing operation
krb5kdc: starting...
W20260808 04:39:21.477140 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.289s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:21 dist-test-slave-drl0 krb5kdc[12221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:39:21.648643 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38109
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:21.801195 12238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:21.801698 12238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:21.801806 12238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:21.807700 12238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:21.807828 12238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:21.807886 12238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:21.807909 12238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:21.807926 12238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:21.814266 12238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38109
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:21.815610 12238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:21.816754 12238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:21.825980 12245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:21.825980 12250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:21.826295 12238 server_base.cc:1034] running on GCE node
W20260808 04:39:21.825968 12246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:21.826975 12238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:21.828281 12238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:21.829475 12238 hybrid_clock.cc:648] HybridClock initialized: now 1786163961829456 us; error 33 us; skew 500 ppm
Aug 08 04:39:21 dist-test-slave-drl0 krb5kdc[12221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:21.832824 12238 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:39:21.833940 12238 webserver.cc:460] Webserver started at http://127.0.0.1:33625/ using document root <none> and password file <none>
I20260808 04:39:21.834558 12238 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:21.834625 12238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:21.834854 12238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:21.837105 12238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "31d5e872215e46edadd8c470be2e68d3"
format_stamp: "Formatted at 2026-08-08 04:39:21 on dist-test-slave-drl0"
server_key: "d967a574a3a1bbd92691efaf687e7f9f"
server_key_iv: "336529bd8a2468147b7d990c0272ea3f"
server_key_version: "encryptionkey@0"
I20260808 04:39:21.837705 12238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "31d5e872215e46edadd8c470be2e68d3"
format_stamp: "Formatted at 2026-08-08 04:39:21 on dist-test-slave-drl0"
server_key: "d967a574a3a1bbd92691efaf687e7f9f"
server_key_iv: "336529bd8a2468147b7d990c0272ea3f"
server_key_version: "encryptionkey@0"
I20260808 04:39:21.842446 12238 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 04:39:21.845916 12260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:21.847457 12238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 04:39:21.847610 12238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "31d5e872215e46edadd8c470be2e68d3"
format_stamp: "Formatted at 2026-08-08 04:39:21 on dist-test-slave-drl0"
server_key: "d967a574a3a1bbd92691efaf687e7f9f"
server_key_iv: "336529bd8a2468147b7d990c0272ea3f"
server_key_version: "encryptionkey@0"
I20260808 04:39:21.847743 12238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:21.863464 12238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:21.868496 12238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:21.868750 12238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:21.877789 12238 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38109
I20260808 04:39:21.877873 12332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38109 every 8 connection(s)
I20260808 04:39:21.879402 12238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:21.882670 12333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:21.888959 12333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3: Bootstrap starting.
I20260808 04:39:21.889480 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 12238
I20260808 04:39:21.889688 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:21.890273 16273 external_mini_cluster.cc:1428] Setting key f34d8f5e898b91f30cbbc585425455b5
I20260808 04:39:21.892058 12333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:21.892902 12333 log.cc:824] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:21.895259 12333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3: No bootstrap required, opened a new log
I20260808 04:39:21.899878 12333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d5e872215e46edadd8c470be2e68d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38109 } }
I20260808 04:39:21.900118 12333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:21.900153 12333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d5e872215e46edadd8c470be2e68d3, State: Initialized, Role: FOLLOWER
I20260808 04:39:21.900693 12333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d5e872215e46edadd8c470be2e68d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38109 } }
I20260808 04:39:21.900895 12333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:21.900986 12333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:21.901119 12333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3 [term 0 FOLLOWER]: Advancing to term 1
Aug 08 04:39:21 dist-test-slave-drl0 krb5kdc[12221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 04:39:21.902413 12333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d5e872215e46edadd8c470be2e68d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38109 } }
I20260808 04:39:21.902792 12333 leader_election.cc:302] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31d5e872215e46edadd8c470be2e68d3; no voters:
I20260808 04:39:21.903203 12333 leader_election.cc:288] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:21.903468 12341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:21.903759 12341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3 [term 1 LEADER]: Becoming Leader. State: Replica: 31d5e872215e46edadd8c470be2e68d3, State: Running, Role: LEADER
I20260808 04:39:21.904134 12341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d5e872215e46edadd8c470be2e68d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38109 } }
I20260808 04:39:21.905009 12333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:21.906203 12345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31d5e872215e46edadd8c470be2e68d3. Latest consensus state: current_term: 1 leader_uuid: "31d5e872215e46edadd8c470be2e68d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d5e872215e46edadd8c470be2e68d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38109 } } }
I20260808 04:39:21.906188 12344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31d5e872215e46edadd8c470be2e68d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d5e872215e46edadd8c470be2e68d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38109 } } }
I20260808 04:39:21.906383 12344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:21.906649 12345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:21.907027 12349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:21.911123 12338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:21.892187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58215 (local address 127.0.0.1:38109)
0808 04:39:21.892872 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:21.892926 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:21.892979 (+ 53us) server_negotiation.cc:407] Connection header received
0808 04:39:21.893905 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:21.893925 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:21.894279 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:21.894783 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:21.895721 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:21.896943 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:21.898608 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:21.899282 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:21.899449 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:21.903162 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:21.903170 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:21.903216 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:21.903269 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:21.906399 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:21.907442 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:21.907449 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:21.907456 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:21.907638 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:21.908154 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:21.908158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:21.908160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:21.908806 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:21.909063 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:21.909694 (+ 631us) server_negotiation.cc:299] Negotiation successful
0808 04:39:21.910056 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":223,"threads_started":1}
I20260808 04:39:21.911180 12349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:21.914889 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:39:21.921381 12349 catalog_manager.cc:1347] Generated new cluster ID: 2f7e073cdb2d41f2a7ac7695a8de54df
I20260808 04:39:21.921461 12349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:21.930132 12338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:21.917604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56307 (local address 127.0.0.1:38109)
0808 04:39:21.917801 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:21.917804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:21.919278 (+ 1474us) server_negotiation.cc:407] Connection header received
0808 04:39:21.919435 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:21.919438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:21.919493 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:21.919580 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:21.920274 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:21.920878 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:21.922120 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:21.922594 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:21.922771 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:21.924583 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:21.924592 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:21.924596 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:21.924758 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:21.926655 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:21.927494 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:21.927498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:21.927500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:21.927802 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:21.928264 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:21.928268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:21.928269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:21.928495 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:21.928629 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:21.929699 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0808 04:39:21.929808 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":104}
I20260808 04:39:21.931011 12349 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:21.932582 12349 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:21.938570 12349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31d5e872215e46edadd8c470be2e68d3: Generated new TSK 0
I20260808 04:39:21.939404 12349 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:21.947444 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 12238
W20260808 04:39:21.964557 12219 connection.cc:535] client connection to 127.0.0.1:38109 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38109): Connection reset by peer (error 104)
2026-08-08T04:39:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1791 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:22 dist-test-slave-drl0 krb5kdc[12379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:22 dist-test-slave-drl0 krb5kdc[12379](info): set up 2 sockets
Aug 08 04:39:22 dist-test-slave-drl0 krb5kdc[12379](info): commencing operation
krb5kdc: starting...
W20260808 04:39:23.230175 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.251s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:23 dist-test-slave-drl0 krb5kdc[12379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:39:23.378648 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:23.508849 12397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:23.509116 12397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:23.509158 12397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:23.513132 12397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:23.513396 12397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:23.513454 12397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:23.513504 12397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:23.513546 12397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:23.519217 12397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43327
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:23.520459 12397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:23.521570 12397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:23.531070 12402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:23.531073 12403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:23.531561 12406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:23.532078 12397 server_base.cc:1034] running on GCE node
I20260808 04:39:23.532776 12397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:23.534049 12397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:23.535465 12397 hybrid_clock.cc:648] HybridClock initialized: now 1786163963535366 us; error 77 us; skew 500 ppm
Aug 08 04:39:23 dist-test-slave-drl0 krb5kdc[12379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:23.543224 12397 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:39:23.544296 12397 webserver.cc:460] Webserver started at http://127.0.0.1:40903/ using document root <none> and password file <none>
I20260808 04:39:23.544902 12397 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:23.545001 12397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:23.545261 12397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:23.547534 12397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "29a689e595c44fedba18d3882c9e7108"
format_stamp: "Formatted at 2026-08-08 04:39:23 on dist-test-slave-drl0"
server_key: "5838e90805c008aba9812a0144810c90"
server_key_iv: "afc33fe81ab18115cc509e34228e7a35"
server_key_version: "encryptionkey@0"
I20260808 04:39:23.548117 12397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "29a689e595c44fedba18d3882c9e7108"
format_stamp: "Formatted at 2026-08-08 04:39:23 on dist-test-slave-drl0"
server_key: "5838e90805c008aba9812a0144810c90"
server_key_iv: "afc33fe81ab18115cc509e34228e7a35"
server_key_version: "encryptionkey@0"
I20260808 04:39:23.552088 12397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:39:23.554832 12414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:23.556401 12397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:39:23.556565 12397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "29a689e595c44fedba18d3882c9e7108"
format_stamp: "Formatted at 2026-08-08 04:39:23 on dist-test-slave-drl0"
server_key: "5838e90805c008aba9812a0144810c90"
server_key_iv: "afc33fe81ab18115cc509e34228e7a35"
server_key_version: "encryptionkey@0"
I20260808 04:39:23.556859 12397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:23.577318 12397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:23.582273 12397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:23.582554 12397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:23.593544 12397 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43327
I20260808 04:39:23.593561 12486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43327 every 8 connection(s)
I20260808 04:39:23.594605 12397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:23.597630 12487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:23.598016 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 12397
I20260808 04:39:23.598205 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:23.598613 16273 external_mini_cluster.cc:1428] Setting key 7212c3222fea228183ab002b6eab26ba
I20260808 04:39:23.603909 12487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108: Bootstrap starting.
I20260808 04:39:23.606127 12487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:23.607065 12487 log.cc:824] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:23.608712 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:23.600533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54285 (local address 127.0.0.1:43327)
0808 04:39:23.601108 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:23.601124 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:23.601165 (+ 41us) server_negotiation.cc:407] Connection header received
0808 04:39:23.601824 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:23.601841 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:23.602180 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:23.602562 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:23.603162 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:23.604195 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:23.605118 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:23.605591 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:23.605762 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:23.606744 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:23.606749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:23.606762 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:23.607259 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:23.607317 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:23.607342 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:23.607481 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:23.607635 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 04:39:23.607780 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":158,"threads_started":1}
I20260808 04:39:23.609314 12487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108: No bootstrap required, opened a new log
I20260808 04:39:23.613147 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:39:23.614368 12487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a689e595c44fedba18d3882c9e7108" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43327 } }
I20260808 04:39:23.614764 12487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:23.614838 12487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29a689e595c44fedba18d3882c9e7108, State: Initialized, Role: FOLLOWER
I20260808 04:39:23.615547 12487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a689e595c44fedba18d3882c9e7108" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43327 } }
I20260808 04:39:23.615851 12487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:23.615948 12487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:23.616091 12487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:23.617599 12487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a689e595c44fedba18d3882c9e7108" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43327 } }
I20260808 04:39:23.618064 12487 leader_election.cc:302] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29a689e595c44fedba18d3882c9e7108; no voters:
I20260808 04:39:23.618454 12487 leader_election.cc:288] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:23.618803 12499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:23.619119 12499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108 [term 1 LEADER]: Becoming Leader. State: Replica: 29a689e595c44fedba18d3882c9e7108, State: Running, Role: LEADER
I20260808 04:39:23.619469 12499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a689e595c44fedba18d3882c9e7108" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43327 } }
I20260808 04:39:23.620029 12487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:23.620754 12499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29a689e595c44fedba18d3882c9e7108. Latest consensus state: current_term: 1 leader_uuid: "29a689e595c44fedba18d3882c9e7108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a689e595c44fedba18d3882c9e7108" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43327 } } }
I20260808 04:39:23.620963 12499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:23.621266 12500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29a689e595c44fedba18d3882c9e7108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29a689e595c44fedba18d3882c9e7108" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43327 } } }
I20260808 04:39:23.621376 12500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:23.621419 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:23.615656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41581 (local address 127.0.0.1:43327)
0808 04:39:23.615797 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:23.615802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:23.616147 (+ 345us) server_negotiation.cc:407] Connection header received
0808 04:39:23.616300 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:23.616302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:23.616347 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:23.616432 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:23.617045 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:23.617808 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:23.619064 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:23.619411 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:23.619494 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:23.620721 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:23.620725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:23.620728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:23.620740 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:23.620769 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:23.620783 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:23.620858 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:23.621201 (+ 343us) server_negotiation.cc:299] Negotiation successful
0808 04:39:23.621246 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 04:39:23.621482 12509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:23.625697 12509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:23.631676 12509 catalog_manager.cc:1347] Generated new cluster ID: f398caa655494841b073a1c67f157358
I20260808 04:39:23.631798 12509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:23.644838 12509 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:23.646001 12509 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:23.655315 12509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29a689e595c44fedba18d3882c9e7108: Generated new TSK 0
I20260808 04:39:23.656140 12509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:23.669342 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 12397
W20260808 04:39:23.681584 12376 connection.cc:535] client connection to 127.0.0.1:43327 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43327): Connection reset by peer (error 104)
2026-08-08T04:39:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1718 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:23 dist-test-slave-drl0 krb5kdc[12534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:23 dist-test-slave-drl0 krb5kdc[12534](info): set up 2 sockets
Aug 08 04:39:23 dist-test-slave-drl0 krb5kdc[12534](info): commencing operation
krb5kdc: starting...
W20260808 04:39:24.969597 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.271s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:25 dist-test-slave-drl0 krb5kdc[12534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:39:25.124966 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33311
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:25.268489 12550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:25.268821 12550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:25.268918 12550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:25.273934 12550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:25.274053 12550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:25.274098 12550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:25.274163 12550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:25.274207 12550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:25.280685 12550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33311
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:25.281891 12550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:25.283375 12550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:25.293051 12558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:25.293102 12560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:25.293332 12550 server_base.cc:1034] running on GCE node
W20260808 04:39:25.293111 12562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:25.294169 12550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:25.295457 12550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:25.296692 12550 hybrid_clock.cc:648] HybridClock initialized: now 1786163965296646 us; error 61 us; skew 500 ppm
Aug 08 04:39:25 dist-test-slave-drl0 krb5kdc[12534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:25.300127 12550 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:39:25.301180 12550 webserver.cc:460] Webserver started at http://127.0.0.1:43067/ using document root <none> and password file <none>
I20260808 04:39:25.301801 12550 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:25.301908 12550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:25.302174 12550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:25.304512 12550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "c2d30f870f6046709a06f9b3247044ad"
format_stamp: "Formatted at 2026-08-08 04:39:25 on dist-test-slave-drl0"
server_key: "859ba0c775f29199dc4710e8fe72e604"
server_key_iv: "97caf0cd7af7d4bd5e7581d68be5ce29"
server_key_version: "encryptionkey@0"
I20260808 04:39:25.305127 12550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "c2d30f870f6046709a06f9b3247044ad"
format_stamp: "Formatted at 2026-08-08 04:39:25 on dist-test-slave-drl0"
server_key: "859ba0c775f29199dc4710e8fe72e604"
server_key_iv: "97caf0cd7af7d4bd5e7581d68be5ce29"
server_key_version: "encryptionkey@0"
I20260808 04:39:25.309656 12550 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:39:25.312999 12569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:25.314513 12550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:39:25.314697 12550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "c2d30f870f6046709a06f9b3247044ad"
format_stamp: "Formatted at 2026-08-08 04:39:25 on dist-test-slave-drl0"
server_key: "859ba0c775f29199dc4710e8fe72e604"
server_key_iv: "97caf0cd7af7d4bd5e7581d68be5ce29"
server_key_version: "encryptionkey@0"
I20260808 04:39:25.314841 12550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:25.320158 12550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:25.323597 12550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:25.323823 12550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:25.332859 12550 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33311
I20260808 04:39:25.332887 12635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33311 every 8 connection(s)
I20260808 04:39:25.333844 12550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:25.334120 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 12550
I20260808 04:39:25.334288 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:25.334673 16273 external_mini_cluster.cc:1428] Setting key afb18aed5fd8bbb3f66d3ac2d458cc2e
I20260808 04:39:25.337240 12637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:25.340500 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:25.336363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43003 (local address 127.0.0.1:33311)
0808 04:39:25.336897 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:25.336913 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:25.336963 (+ 50us) server_negotiation.cc:407] Connection header received
0808 04:39:25.337794 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:25.337812 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:25.338091 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:25.338444 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:25.339229 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:25.339233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:25.339245 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:25.339255 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:25.339308 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:25.339327 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:25.339492 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:25.339562 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 04:39:25.339715 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":158,"threads_started":1}
I20260808 04:39:25.344278 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:39:25.344734 12637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad: Bootstrap starting.
I20260808 04:39:25.348162 12637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:25.348737 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:25.346716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45599 (local address 127.0.0.1:33311)
0808 04:39:25.346884 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:25.346889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:25.346914 (+ 25us) server_negotiation.cc:407] Connection header received
0808 04:39:25.347194 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:25.347199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:25.347250 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:25.347360 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:25.348249 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:25.348283 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:25.348289 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:25.348303 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:25.348334 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:25.348383 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:25.348484 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:25.348606 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 04:39:25.348642 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:39:25.349128 12637 log.cc:824] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:25.351395 12637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad: No bootstrap required, opened a new log
I20260808 04:39:25.355878 12637 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d30f870f6046709a06f9b3247044ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33311 } }
I20260808 04:39:25.356125 12637 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:25.356173 12637 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d30f870f6046709a06f9b3247044ad, State: Initialized, Role: FOLLOWER
I20260808 04:39:25.356542 12637 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d30f870f6046709a06f9b3247044ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33311 } }
I20260808 04:39:25.356666 12637 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:25.356709 12637 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:25.356786 12637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:25.357692 12637 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d30f870f6046709a06f9b3247044ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33311 } }
I20260808 04:39:25.358024 12637 leader_election.cc:302] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2d30f870f6046709a06f9b3247044ad; no voters:
I20260808 04:39:25.358353 12637 leader_election.cc:288] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:25.358409 12647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:25.358618 12647 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad [term 1 LEADER]: Becoming Leader. State: Replica: c2d30f870f6046709a06f9b3247044ad, State: Running, Role: LEADER
I20260808 04:39:25.358912 12647 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d30f870f6046709a06f9b3247044ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33311 } }
I20260808 04:39:25.359732 12637 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:25.361029 12649 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2d30f870f6046709a06f9b3247044ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d30f870f6046709a06f9b3247044ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33311 } } }
I20260808 04:39:25.361172 12649 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:25.361732 12660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:25.362424 12652 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2d30f870f6046709a06f9b3247044ad. Latest consensus state: current_term: 1 leader_uuid: "c2d30f870f6046709a06f9b3247044ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d30f870f6046709a06f9b3247044ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33311 } } }
I20260808 04:39:25.362540 12652 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:25.365584 12660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:25.372320 12660 catalog_manager.cc:1347] Generated new cluster ID: 24b8795b536946f089e0a87525df7da3
I20260808 04:39:25.372419 12660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:25.385145 12660 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:25.385774 12660 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:25.401351 12660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2d30f870f6046709a06f9b3247044ad: Generated new TSK 0
I20260808 04:39:25.402248 12660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:25.409298 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 12550
2026-08-08T04:39:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1746 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:25 dist-test-slave-drl0 krb5kdc[12677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:25 dist-test-slave-drl0 krb5kdc[12677](info): set up 2 sockets
Aug 08 04:39:25 dist-test-slave-drl0 krb5kdc[12677](info): commencing operation
krb5kdc: starting...
W20260808 04:39:26.715804 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.272s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:26 dist-test-slave-drl0 krb5kdc[12677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:39:26.867425 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42555
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:27.000054 12698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:27.000433 12698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:27.000533 12698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:27.005650 12698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:27.005781 12698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:27.005831 12698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:27.005874 12698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:27.005909 12698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:27.012352 12698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:27.013590 12698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:27.014683 12698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 04:39:27.025326 12698 server_base.cc:1034] running on GCE node
W20260808 04:39:27.025374 12709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:27.025358 12708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:27.025384 12711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:27.026340 12698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:27.027494 12698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:27.028694 12698 hybrid_clock.cc:648] HybridClock initialized: now 1786163967028650 us; error 54 us; skew 500 ppm
Aug 08 04:39:27 dist-test-slave-drl0 krb5kdc[12677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:27.031817 12698 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:39:27.032727 12698 webserver.cc:460] Webserver started at http://127.0.0.1:33343/ using document root <none> and password file <none>
I20260808 04:39:27.033299 12698 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:27.033371 12698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:27.033588 12698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:27.035758 12698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "983c1b4e81d04eb8a3312b72fc48a1ae"
format_stamp: "Formatted at 2026-08-08 04:39:27 on dist-test-slave-drl0"
server_key: "3feb8e4df619a9fd02e55a980002af99"
server_key_iv: "a33476e938887fda286025eda0141c6b"
server_key_version: "encryptionkey@0"
I20260808 04:39:27.036338 12698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "983c1b4e81d04eb8a3312b72fc48a1ae"
format_stamp: "Formatted at 2026-08-08 04:39:27 on dist-test-slave-drl0"
server_key: "3feb8e4df619a9fd02e55a980002af99"
server_key_iv: "a33476e938887fda286025eda0141c6b"
server_key_version: "encryptionkey@0"
I20260808 04:39:27.040309 12698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:39:27.043078 12725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:27.044512 12698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:39:27.044624 12698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "983c1b4e81d04eb8a3312b72fc48a1ae"
format_stamp: "Formatted at 2026-08-08 04:39:27 on dist-test-slave-drl0"
server_key: "3feb8e4df619a9fd02e55a980002af99"
server_key_iv: "a33476e938887fda286025eda0141c6b"
server_key_version: "encryptionkey@0"
I20260808 04:39:27.044806 12698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:27.054643 12698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:27.057924 12698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:27.058085 12698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:27.066164 12698 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42555
I20260808 04:39:27.066524 12792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42555 every 8 connection(s)
I20260808 04:39:27.067294 12698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:27.070385 12793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:27.076980 12793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae: Bootstrap starting.
I20260808 04:39:27.077128 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 12698
I20260808 04:39:27.077267 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:27.077580 16273 external_mini_cluster.cc:1428] Setting key 15c1a467dc3383d728cf70b22a2885b3
I20260808 04:39:27.079977 12793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:27.080749 12793 log.cc:824] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:27.082762 12793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae: No bootstrap required, opened a new log
I20260808 04:39:27.086292 12793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983c1b4e81d04eb8a3312b72fc48a1ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42555 } }
I20260808 04:39:27.086508 12793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:27.086542 12793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983c1b4e81d04eb8a3312b72fc48a1ae, State: Initialized, Role: FOLLOWER
Aug 08 04:39:27 dist-test-slave-drl0 krb5kdc[12677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 04:39:27.087380 12793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983c1b4e81d04eb8a3312b72fc48a1ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42555 } }
I20260808 04:39:27.087591 12793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:27.087697 12793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:27.087845 12793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:27.089321 12793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983c1b4e81d04eb8a3312b72fc48a1ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42555 } }
I20260808 04:39:27.089748 12793 leader_election.cc:302] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 983c1b4e81d04eb8a3312b72fc48a1ae; no voters:
I20260808 04:39:27.090163 12793 leader_election.cc:288] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:27.090323 12800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:27.090587 12800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae [term 1 LEADER]: Becoming Leader. State: Replica: 983c1b4e81d04eb8a3312b72fc48a1ae, State: Running, Role: LEADER
I20260808 04:39:27.090905 12800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983c1b4e81d04eb8a3312b72fc48a1ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42555 } }
I20260808 04:39:27.091708 12793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:27.092765 12801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "983c1b4e81d04eb8a3312b72fc48a1ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983c1b4e81d04eb8a3312b72fc48a1ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42555 } } }
I20260808 04:39:27.092908 12801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:27.093271 12802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 983c1b4e81d04eb8a3312b72fc48a1ae. Latest consensus state: current_term: 1 leader_uuid: "983c1b4e81d04eb8a3312b72fc48a1ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983c1b4e81d04eb8a3312b72fc48a1ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42555 } } }
I20260808 04:39:27.093384 12802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:27.093689 12814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:27.094892 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:27.079429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52693 (local address 127.0.0.1:42555)
0808 04:39:27.079956 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:27.079972 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:27.080013 (+ 41us) server_negotiation.cc:407] Connection header received
0808 04:39:27.080760 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:27.080777 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:27.081194 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:27.081565 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:27.082510 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:27.083622 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:27.084504 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:27.084999 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:27.085174 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:27.087940 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:27.087948 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:27.087967 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:27.088003 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:27.090339 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:27.090954 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:27.090961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:27.090967 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:27.091112 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:27.091554 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:27.091558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:27.091560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:27.092124 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:27.093220 (+ 1096us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:27.093397 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 04:39:27.093773 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":164,"threads_started":1}
I20260808 04:39:27.096767 12814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:27.098517 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:39:27.104395 12814 catalog_manager.cc:1347] Generated new cluster ID: 9f0cd25e8bf144f591422d5b1721d269
I20260808 04:39:27.104468 12814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:27.111912 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:27.101941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57395 (local address 127.0.0.1:42555)
0808 04:39:27.102073 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:27.102076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:27.102585 (+ 509us) server_negotiation.cc:407] Connection header received
0808 04:39:27.102638 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:27.102640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:27.102695 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:27.102780 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:27.103860 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:27.104778 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:27.105983 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:27.106357 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:27.106460 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:27.108285 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:27.108289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:27.108291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:27.108317 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:27.110003 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:27.110498 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:27.110501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:27.110503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:27.110581 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:27.110881 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:27.110883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:27.110885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:27.111104 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:27.111252 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:27.111630 (+ 378us) server_negotiation.cc:299] Negotiation successful
0808 04:39:27.111752 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 04:39:27.141996 12814 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:27.143126 12814 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:27.152871 12814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 983c1b4e81d04eb8a3312b72fc48a1ae: Generated new TSK 0
I20260808 04:39:27.153589 12814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:27.163792 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 12698
2026-08-08T04:39:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1752 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:27 dist-test-slave-drl0 krb5kdc[12838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:27 dist-test-slave-drl0 krb5kdc[12838](info): set up 2 sockets
Aug 08 04:39:27 dist-test-slave-drl0 krb5kdc[12838](info): commencing operation
krb5kdc: starting...
W20260808 04:39:28.443758 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.247s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:28 dist-test-slave-drl0 krb5kdc[12838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:39:28.596958 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:28.723482 12858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:28.723788 12858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:28.723841 12858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:28.728609 12858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:28.728749 12858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:28.728772 12858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:28.728791 12858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:28.728811 12858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:28.737263 12858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:28.738729 12858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:28.739938 12858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:28.751259 12872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:28.751260 12867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:28.751497 12869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:28.751793 12858 server_base.cc:1034] running on GCE node
I20260808 04:39:28.752442 12858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:28.753783 12858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:28.755133 12858 hybrid_clock.cc:648] HybridClock initialized: now 1786163968755128 us; error 52 us; skew 500 ppm
Aug 08 04:39:28 dist-test-slave-drl0 krb5kdc[12838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:28.761792 12858 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:39:28.763329 12858 webserver.cc:460] Webserver started at http://127.0.0.1:45035/ using document root <none> and password file <none>
I20260808 04:39:28.764091 12858 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:28.764210 12858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:28.764509 12858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:28.767488 12858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "4688a43752d74ebf9bc000abc4ca48f5"
format_stamp: "Formatted at 2026-08-08 04:39:28 on dist-test-slave-drl0"
server_key: "362717d9bcf5f1fc36de347dc102fe0d"
server_key_iv: "fe3fb07c2182449a3dda81a859a69c22"
server_key_version: "encryptionkey@0"
I20260808 04:39:28.768177 12858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "4688a43752d74ebf9bc000abc4ca48f5"
format_stamp: "Formatted at 2026-08-08 04:39:28 on dist-test-slave-drl0"
server_key: "362717d9bcf5f1fc36de347dc102fe0d"
server_key_iv: "fe3fb07c2182449a3dda81a859a69c22"
server_key_version: "encryptionkey@0"
I20260808 04:39:28.772316 12858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:39:28.775520 12880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:28.777546 12858 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260808 04:39:28.777798 12858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "4688a43752d74ebf9bc000abc4ca48f5"
format_stamp: "Formatted at 2026-08-08 04:39:28 on dist-test-slave-drl0"
server_key: "362717d9bcf5f1fc36de347dc102fe0d"
server_key_iv: "fe3fb07c2182449a3dda81a859a69c22"
server_key_version: "encryptionkey@0"
I20260808 04:39:28.777976 12858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:28.802508 12858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:28.807474 12858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:28.807763 12858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:28.816030 12858 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36539
I20260808 04:39:28.816042 12952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36539 every 8 connection(s)
I20260808 04:39:28.817103 12858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:28.820171 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:28.825878 12955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5: Bootstrap starting.
I20260808 04:39:28.826721 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 12858
I20260808 04:39:28.826901 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:28.827231 16273 external_mini_cluster.cc:1428] Setting key 1c0d3df396dfdbd61cf41e57eb28d427
I20260808 04:39:28.829509 12955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:28.830529 12955 log.cc:824] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:28.833004 12955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5: No bootstrap required, opened a new log
I20260808 04:39:28.837791 12959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:28.828732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58603 (local address 127.0.0.1:36539)
0808 04:39:28.829365 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:28.829383 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:28.829421 (+ 38us) server_negotiation.cc:407] Connection header received
0808 04:39:28.830253 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:28.830273 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:28.830632 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:28.831047 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:28.831498 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:28.832602 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:28.834175 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:28.834690 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:28.834889 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:28.835986 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:28.835992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:28.836005 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:28.836428 (+ 423us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:28.836489 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:28.836513 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:28.836636 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:28.836762 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 04:39:28.836999 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":206,"threads_started":1}
I20260808 04:39:28.837815 12955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4688a43752d74ebf9bc000abc4ca48f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36539 } }
I20260808 04:39:28.838218 12955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:28.838315 12955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4688a43752d74ebf9bc000abc4ca48f5, State: Initialized, Role: FOLLOWER
I20260808 04:39:28.838842 12955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4688a43752d74ebf9bc000abc4ca48f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36539 } }
I20260808 04:39:28.839047 12955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:28.839155 12955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:28.839272 12955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:28.840420 12955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4688a43752d74ebf9bc000abc4ca48f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36539 } }
I20260808 04:39:28.840811 12955 leader_election.cc:302] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4688a43752d74ebf9bc000abc4ca48f5; no voters:
I20260808 04:39:28.841233 12955 leader_election.cc:288] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:28.841418 12962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:28.841953 12962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4688a43752d74ebf9bc000abc4ca48f5, State: Running, Role: LEADER
I20260808 04:39:28.842028 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:39:28.842474 12962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4688a43752d74ebf9bc000abc4ca48f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36539 } }
I20260808 04:39:28.842794 12955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:28.845449 12965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4688a43752d74ebf9bc000abc4ca48f5. Latest consensus state: current_term: 1 leader_uuid: "4688a43752d74ebf9bc000abc4ca48f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4688a43752d74ebf9bc000abc4ca48f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36539 } } }
I20260808 04:39:28.845626 12965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:28.845968 12963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4688a43752d74ebf9bc000abc4ca48f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4688a43752d74ebf9bc000abc4ca48f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36539 } } }
I20260808 04:39:28.846140 12963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:28.850121 12959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:28.844835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55465 (local address 127.0.0.1:36539)
0808 04:39:28.844988 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:28.844994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:28.845068 (+ 74us) server_negotiation.cc:407] Connection header received
0808 04:39:28.845125 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:28.845131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:28.845195 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:28.845291 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:28.845665 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:28.846453 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:28.848004 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:28.848262 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:28.848386 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:28.849547 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:28.849552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:28.849555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:28.849573 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:28.849615 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:28.849631 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:28.849720 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:28.849870 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 04:39:28.849951 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260808 04:39:28.851262 12992 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:39:28.851423 12992 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:39:28.851568 12987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:28.852394 12987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:28.858291 12987 catalog_manager.cc:1347] Generated new cluster ID: 8d496480bf534a8faec06c70109a77d8
I20260808 04:39:28.858388 12987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:28.869539 12987 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:28.870841 12987 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:28.878159 12987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4688a43752d74ebf9bc000abc4ca48f5: Generated new TSK 0
I20260808 04:39:28.878990 12987 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:28.886277 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 12858
2026-08-08T04:39:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1719 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:28 dist-test-slave-drl0 krb5kdc[13003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:28 dist-test-slave-drl0 krb5kdc[13003](info): set up 2 sockets
Aug 08 04:39:28 dist-test-slave-drl0 krb5kdc[13003](info): commencing operation
krb5kdc: starting...
W20260808 04:39:30.153084 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:30 dist-test-slave-drl0 krb5kdc[13003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:39:30.303469 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35095
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:30.435359 13022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:30.435629 13022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:30.435672 13022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:30.439719 13022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:30.439785 13022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:30.439806 13022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:30.439823 13022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:30.439838 13022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:30.445227 13022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:30.446305 13022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:30.447376 13022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:30.456641 13030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:30.456696 13028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:30.456816 13022 server_base.cc:1034] running on GCE node
W20260808 04:39:30.456637 13033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:30.457609 13022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:30.458765 13022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:30.460145 13022 hybrid_clock.cc:648] HybridClock initialized: now 1786163970460113 us; error 45 us; skew 500 ppm
Aug 08 04:39:30 dist-test-slave-drl0 krb5kdc[13003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:30.464154 13022 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:39:30.465307 13022 webserver.cc:460] Webserver started at http://127.0.0.1:42071/ using document root <none> and password file <none>
I20260808 04:39:30.465886 13022 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:30.465952 13022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:30.466126 13022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:30.468472 13022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "afd627c2dc83407d8eca2958cfb3d3a8"
format_stamp: "Formatted at 2026-08-08 04:39:30 on dist-test-slave-drl0"
server_key: "fbef66345d12c35982ac2a6faf7a2418"
server_key_iv: "359993feff6dc8dcd14301a173745cf0"
server_key_version: "encryptionkey@0"
I20260808 04:39:30.469029 13022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "afd627c2dc83407d8eca2958cfb3d3a8"
format_stamp: "Formatted at 2026-08-08 04:39:30 on dist-test-slave-drl0"
server_key: "fbef66345d12c35982ac2a6faf7a2418"
server_key_iv: "359993feff6dc8dcd14301a173745cf0"
server_key_version: "encryptionkey@0"
I20260808 04:39:30.473186 13022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:39:30.476019 13043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:30.477766 13022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:39:30.477885 13022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "afd627c2dc83407d8eca2958cfb3d3a8"
format_stamp: "Formatted at 2026-08-08 04:39:30 on dist-test-slave-drl0"
server_key: "fbef66345d12c35982ac2a6faf7a2418"
server_key_iv: "359993feff6dc8dcd14301a173745cf0"
server_key_version: "encryptionkey@0"
I20260808 04:39:30.478055 13022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:30.484333 13022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:30.488161 13022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:30.488340 13022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:30.498710 13022 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35095
I20260808 04:39:30.498713 13112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35095 every 8 connection(s)
I20260808 04:39:30.500177 13022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:30.503381 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 13022
I20260808 04:39:30.503559 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:30.503878 16273 external_mini_cluster.cc:1428] Setting key d1c54c1e7738e973a886004585500e32
I20260808 04:39:30.504025 13113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:30.510605 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:30.505494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51447 (local address 127.0.0.1:35095)
0808 04:39:30.506005 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:30.506026 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:30.506072 (+ 46us) server_negotiation.cc:407] Connection header received
0808 04:39:30.507329 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:30.507357 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:30.507773 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:30.508185 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:30.509011 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:30.509018 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:30.509033 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:30.509047 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:30.509119 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:30.509148 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:30.509309 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:30.509423 (+ 114us) server_negotiation.cc:299] Negotiation successful
0808 04:39:30.509593 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":129,"threads_started":1}
I20260808 04:39:30.512750 13113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8: Bootstrap starting.
I20260808 04:39:30.515143 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:39:30.515558 13113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:30.516510 13113 log.cc:824] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:30.518967 13113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8: No bootstrap required, opened a new log
I20260808 04:39:30.519997 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:30.517635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48501 (local address 127.0.0.1:35095)
0808 04:39:30.517771 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:30.517777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:30.517925 (+ 148us) server_negotiation.cc:407] Connection header received
0808 04:39:30.518031 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:30.518034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:30.518069 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:30.518159 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:30.519506 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:30.519510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:30.519513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:30.519526 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:30.519555 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:30.519569 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:30.519673 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:30.519832 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 04:39:30.519874 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:39:30.522912 13113 raft_consensus.cc:348] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd627c2dc83407d8eca2958cfb3d3a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35095 } }
I20260808 04:39:30.523253 13113 raft_consensus.cc:374] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:30.523337 13113 raft_consensus.cc:729] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afd627c2dc83407d8eca2958cfb3d3a8, State: Initialized, Role: FOLLOWER
I20260808 04:39:30.523813 13113 consensus_queue.cc:260] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd627c2dc83407d8eca2958cfb3d3a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35095 } }
I20260808 04:39:30.523968 13113 raft_consensus.cc:388] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:30.524042 13113 raft_consensus.cc:482] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:30.524143 13113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:30.525789 13113 raft_consensus.cc:504] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd627c2dc83407d8eca2958cfb3d3a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35095 } }
I20260808 04:39:30.526437 13113 leader_election.cc:302] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afd627c2dc83407d8eca2958cfb3d3a8; no voters:
I20260808 04:39:30.526911 13113 leader_election.cc:288] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:30.527145 13129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:30.527534 13129 raft_consensus.cc:686] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8 [term 1 LEADER]: Becoming Leader. State: Replica: afd627c2dc83407d8eca2958cfb3d3a8, State: Running, Role: LEADER
I20260808 04:39:30.527940 13129 consensus_queue.cc:237] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd627c2dc83407d8eca2958cfb3d3a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35095 } }
I20260808 04:39:30.528942 13113 sys_catalog.cc:564] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:30.529850 13130 sys_catalog.cc:455] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afd627c2dc83407d8eca2958cfb3d3a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd627c2dc83407d8eca2958cfb3d3a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35095 } } }
I20260808 04:39:30.530027 13130 sys_catalog.cc:458] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:30.529850 13131 sys_catalog.cc:455] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afd627c2dc83407d8eca2958cfb3d3a8. Latest consensus state: current_term: 1 leader_uuid: "afd627c2dc83407d8eca2958cfb3d3a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd627c2dc83407d8eca2958cfb3d3a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35095 } } }
I20260808 04:39:30.530385 13131 sys_catalog.cc:458] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:30.530480 13137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:30.534152 13137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:30.541044 13137 catalog_manager.cc:1347] Generated new cluster ID: 97db120c78d84e2dbced8f46987ca2fa
I20260808 04:39:30.541123 13137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:30.579885 13137 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:30.580274 13137 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:30.603459 13137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afd627c2dc83407d8eca2958cfb3d3a8: Generated new TSK 0
I20260808 04:39:30.604643 13137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:30.616587 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 13022
2026-08-08T04:39:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1729 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:30 dist-test-slave-drl0 krb5kdc[13160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:30 dist-test-slave-drl0 krb5kdc[13160](info): set up 2 sockets
Aug 08 04:39:30 dist-test-slave-drl0 krb5kdc[13160](info): commencing operation
krb5kdc: starting...
W20260808 04:39:31.900388 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.005s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:32 dist-test-slave-drl0 krb5kdc[13160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:39:32.051017 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46061
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:32.181494 13180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:32.181819 13180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:32.181870 13180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:32.186578 13180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:32.186655 13180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:32.186676 13180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:32.186693 13180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:32.186709 13180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:32.192742 13180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:32.193933 13180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:32.195042 13180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:32.204262 13191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:32.204339 13189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:32.204535 13180 server_base.cc:1034] running on GCE node
W20260808 04:39:32.204272 13188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:32.205492 13180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:32.206663 13180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:32.207886 13180 hybrid_clock.cc:648] HybridClock initialized: now 1786163972207877 us; error 64 us; skew 500 ppm
Aug 08 04:39:32 dist-test-slave-drl0 krb5kdc[13160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:32.211221 13180 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:39:32.212157 13180 webserver.cc:460] Webserver started at http://127.0.0.1:46345/ using document root <none> and password file <none>
I20260808 04:39:32.212738 13180 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:32.212842 13180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:32.213081 13180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:32.215351 13180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "6c4903f5fd824362ad904f9a1b067ab3"
format_stamp: "Formatted at 2026-08-08 04:39:32 on dist-test-slave-drl0"
server_key: "1202aad1049aa6e642e674538badb1bb"
server_key_iv: "3b155cb7332512b53e86fc1d8f622b98"
server_key_version: "encryptionkey@0"
I20260808 04:39:32.215943 13180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "6c4903f5fd824362ad904f9a1b067ab3"
format_stamp: "Formatted at 2026-08-08 04:39:32 on dist-test-slave-drl0"
server_key: "1202aad1049aa6e642e674538badb1bb"
server_key_iv: "3b155cb7332512b53e86fc1d8f622b98"
server_key_version: "encryptionkey@0"
I20260808 04:39:32.220427 13180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:39:32.224220 13200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:32.225781 13180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260808 04:39:32.225988 13180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "6c4903f5fd824362ad904f9a1b067ab3"
format_stamp: "Formatted at 2026-08-08 04:39:32 on dist-test-slave-drl0"
server_key: "1202aad1049aa6e642e674538badb1bb"
server_key_iv: "3b155cb7332512b53e86fc1d8f622b98"
server_key_version: "encryptionkey@0"
I20260808 04:39:32.226145 13180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:32.264181 13180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:32.267932 13180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:32.268177 13180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:32.277957 13180 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46061
I20260808 04:39:32.277997 13269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46061 every 8 connection(s)
I20260808 04:39:32.279222 13180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:32.280846 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 13180
I20260808 04:39:32.281068 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:32.281648 16273 external_mini_cluster.cc:1428] Setting key 382880fb2eb08ccc68cc5e79a1879b91
I20260808 04:39:32.283768 13271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:32.290683 13271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3: Bootstrap starting.
Aug 08 04:39:32 dist-test-slave-drl0 krb5kdc[13160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 04:39:32.293414 13271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:32.294479 13271 log.cc:824] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:32.297194 13271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3: No bootstrap required, opened a new log
I20260808 04:39:32.298656 13275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:32.283430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:35607 (local address 127.0.0.1:46061)
0808 04:39:32.283967 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:32.283987 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:32.284032 (+ 45us) server_negotiation.cc:407] Connection header received
0808 04:39:32.284990 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:32.285014 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:32.285506 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:32.285837 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:32.286680 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:32.287766 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:32.288633 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:32.289106 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:32.289273 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:32.292205 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:32.292211 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:32.292225 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:32.292259 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:32.294708 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:32.295594 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:32.295600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:32.295607 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:32.295761 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:32.296191 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:32.296194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:32.296196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:32.296781 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:32.297035 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:32.297652 (+ 617us) server_negotiation.cc:299] Negotiation successful
0808 04:39:32.297886 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":151,"threads_started":1}
I20260808 04:39:32.301493 13271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4903f5fd824362ad904f9a1b067ab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46061 } }
I20260808 04:39:32.301755 13271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:32.301826 13271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c4903f5fd824362ad904f9a1b067ab3, State: Initialized, Role: FOLLOWER
I20260808 04:39:32.302395 13271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4903f5fd824362ad904f9a1b067ab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46061 } }
I20260808 04:39:32.302500 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:39:32.302538 13271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:32.302634 13271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:32.302762 13271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:32.303894 13271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4903f5fd824362ad904f9a1b067ab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46061 } }
I20260808 04:39:32.304293 13271 leader_election.cc:302] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c4903f5fd824362ad904f9a1b067ab3; no voters:
I20260808 04:39:32.304664 13271 leader_election.cc:288] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:32.304895 13283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:32.305224 13283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3 [term 1 LEADER]: Becoming Leader. State: Replica: 6c4903f5fd824362ad904f9a1b067ab3, State: Running, Role: LEADER
I20260808 04:39:32.305599 13283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4903f5fd824362ad904f9a1b067ab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46061 } }
I20260808 04:39:32.306098 13271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:32.307611 13285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c4903f5fd824362ad904f9a1b067ab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4903f5fd824362ad904f9a1b067ab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46061 } } }
I20260808 04:39:32.307770 13285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:32.308197 13301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:32.308178 13287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c4903f5fd824362ad904f9a1b067ab3. Latest consensus state: current_term: 1 leader_uuid: "6c4903f5fd824362ad904f9a1b067ab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4903f5fd824362ad904f9a1b067ab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46061 } } }
I20260808 04:39:32.310165 13287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:32.312472 13301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:32.315984 13275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:32.304709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:57317 (local address 127.0.0.1:46061)
0808 04:39:32.304946 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:32.304953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:32.305005 (+ 52us) server_negotiation.cc:407] Connection header received
0808 04:39:32.305186 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:32.305189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:32.305234 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:32.305288 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:32.306146 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:32.306958 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:32.308424 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:32.308835 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:32.308932 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:32.311254 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:32.311260 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:32.311263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:32.311295 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:32.313348 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:32.314111 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:32.314117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:32.314119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:32.314228 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:32.314770 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:32.314774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:32.314777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:32.315084 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:32.315204 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:32.315652 (+ 448us) server_negotiation.cc:299] Negotiation successful
0808 04:39:32.315778 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260808 04:39:32.318915 13301 catalog_manager.cc:1347] Generated new cluster ID: 66664723094b48d1b9b0c59cc062f9d7
I20260808 04:39:32.319010 13301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:32.334698 13301 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:32.335850 13301 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:32.343873 13301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c4903f5fd824362ad904f9a1b067ab3: Generated new TSK 0
I20260808 04:39:32.344871 13301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:32.353436 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 13180
W20260808 04:39:32.365835 13154 connection.cc:535] client connection to 127.0.0.1:46061 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46061): Connection reset by peer (error 104)
2026-08-08T04:39:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1734 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:32 dist-test-slave-drl0 krb5kdc[13319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:32 dist-test-slave-drl0 krb5kdc[13319](info): set up 2 sockets
Aug 08 04:39:32 dist-test-slave-drl0 krb5kdc[13319](info): commencing operation
krb5kdc: starting...
W20260808 04:39:33.666742 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.286s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:33 dist-test-slave-drl0 krb5kdc[13319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:39:33.815384 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43261
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:33.938898 13335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:33.939241 13335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:33.939288 13335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:33.943514 13335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:33.943576 13335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:33.943596 13335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:33.943614 13335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:33.943629 13335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:33.950532 13335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:33.952185 13335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:33.953694 13335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:33.963846 13343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:33.963822 13341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:33.963819 13346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:33.964265 13335 server_base.cc:1034] running on GCE node
I20260808 04:39:33.964883 13335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:33.966147 13335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:33.967440 13335 hybrid_clock.cc:648] HybridClock initialized: now 1786163973967407 us; error 38 us; skew 500 ppm
Aug 08 04:39:33 dist-test-slave-drl0 krb5kdc[13319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:33.976122 13335 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:39:33.977778 13335 webserver.cc:460] Webserver started at http://127.0.0.1:34791/ using document root <none> and password file <none>
I20260808 04:39:33.978627 13335 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:33.978752 13335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:33.979127 13335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:33.982160 13335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "932cbe47dc034f7eb7954f5ba92c30d1"
format_stamp: "Formatted at 2026-08-08 04:39:33 on dist-test-slave-drl0"
server_key: "2a6c7adf5bd84ec7f8baac989e63111d"
server_key_iv: "b7a9230a597d6d250e9acd4fb0ca006b"
server_key_version: "encryptionkey@0"
I20260808 04:39:33.983403 13335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "932cbe47dc034f7eb7954f5ba92c30d1"
format_stamp: "Formatted at 2026-08-08 04:39:33 on dist-test-slave-drl0"
server_key: "2a6c7adf5bd84ec7f8baac989e63111d"
server_key_iv: "b7a9230a597d6d250e9acd4fb0ca006b"
server_key_version: "encryptionkey@0"
I20260808 04:39:33.987829 13335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:39:33.990504 13355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:33.992601 13335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:39:33.992738 13335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "932cbe47dc034f7eb7954f5ba92c30d1"
format_stamp: "Formatted at 2026-08-08 04:39:33 on dist-test-slave-drl0"
server_key: "2a6c7adf5bd84ec7f8baac989e63111d"
server_key_iv: "b7a9230a597d6d250e9acd4fb0ca006b"
server_key_version: "encryptionkey@0"
I20260808 04:39:33.992884 13335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:34.003998 13335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:34.007795 13335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:34.008008 13335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:34.016047 13335 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43261
I20260808 04:39:34.016077 13424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43261 every 8 connection(s)
I20260808 04:39:34.017026 13335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:34.019843 13425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:34.024871 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 13335
I20260808 04:39:34.025048 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:34.025359 16273 external_mini_cluster.cc:1428] Setting key 004650f571f264edd29086b2b4493b37
I20260808 04:39:34.026196 13425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1: Bootstrap starting.
I20260808 04:39:34.028748 13425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:34.029676 13425 log.cc:824] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:34.031826 13425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1: No bootstrap required, opened a new log
I20260808 04:39:34.035894 13425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932cbe47dc034f7eb7954f5ba92c30d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43261 } }
I20260808 04:39:34.036161 13425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:34.036253 13425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 932cbe47dc034f7eb7954f5ba92c30d1, State: Initialized, Role: FOLLOWER
I20260808 04:39:34.036762 13425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932cbe47dc034f7eb7954f5ba92c30d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43261 } }
I20260808 04:39:34.036818 13428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:34.026904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:43313 (local address 127.0.0.1:43261)
0808 04:39:34.027500 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:34.027522 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:34.027561 (+ 39us) server_negotiation.cc:407] Connection header received
0808 04:39:34.028232 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:34.028248 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:34.028591 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:34.028944 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:34.029399 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:34.030636 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:34.031604 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:34.032122 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:34.032297 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:34.033379 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:34.033384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:34.033396 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:34.034373 (+ 977us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:34.034525 (+ 152us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:34.034577 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:34.035034 (+ 457us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:34.035313 (+ 279us) server_negotiation.cc:299] Negotiation successful
0808 04:39:34.035547 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":134,"threads_started":1}
I20260808 04:39:34.036942 13425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:34.037225 13425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:34.037449 13425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:34.038761 13425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932cbe47dc034f7eb7954f5ba92c30d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43261 } }
I20260808 04:39:34.039147 13425 leader_election.cc:302] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 932cbe47dc034f7eb7954f5ba92c30d1; no voters:
I20260808 04:39:34.039502 13425 leader_election.cc:288] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:34.039613 13430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:34.039876 13430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1 [term 1 LEADER]: Becoming Leader. State: Replica: 932cbe47dc034f7eb7954f5ba92c30d1, State: Running, Role: LEADER
I20260808 04:39:34.040263 13430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932cbe47dc034f7eb7954f5ba92c30d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43261 } }
I20260808 04:39:34.040812 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:39:34.040903 13425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:34.042109 13431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "932cbe47dc034f7eb7954f5ba92c30d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932cbe47dc034f7eb7954f5ba92c30d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43261 } } }
I20260808 04:39:34.042297 13431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:34.043124 13432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 932cbe47dc034f7eb7954f5ba92c30d1. Latest consensus state: current_term: 1 leader_uuid: "932cbe47dc034f7eb7954f5ba92c30d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932cbe47dc034f7eb7954f5ba92c30d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43261 } } }
I20260808 04:39:34.043505 13432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1 [sys.catalog]: This master's current role is: LEADER
W20260808 04:39:34.046607 13456 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:39:34.046753 13456 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:39:34.046883 13446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:34.047874 13446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:34.049010 13428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:34.043781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:44363 (local address 127.0.0.1:43261)
0808 04:39:34.043899 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:34.043906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:34.044053 (+ 147us) server_negotiation.cc:407] Connection header received
0808 04:39:34.044266 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:34.044269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:34.044327 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:34.044424 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:34.044858 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:34.045629 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:34.046755 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:34.047188 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:34.047325 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:34.048422 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:34.048426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:34.048430 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:34.048456 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:34.048487 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:34.048499 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:34.048607 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:34.048775 (+ 168us) server_negotiation.cc:299] Negotiation successful
0808 04:39:34.048825 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 04:39:34.054862 13446 catalog_manager.cc:1347] Generated new cluster ID: cfdbf52cab2a479aaef53a14edacc7d1
I20260808 04:39:34.055009 13446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:34.069799 13446 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:34.070803 13446 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:34.083803 13446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 932cbe47dc034f7eb7954f5ba92c30d1: Generated new TSK 0
I20260808 04:39:34.084594 13446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:34.092151 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 13335
W20260808 04:39:34.106530 13317 connection.cc:535] client connection to 127.0.0.1:43261 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43261): Connection reset by peer (error 104)
2026-08-08T04:39:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1740 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:34 dist-test-slave-drl0 krb5kdc[13467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:34 dist-test-slave-drl0 krb5kdc[13467](info): set up 2 sockets
Aug 08 04:39:34 dist-test-slave-drl0 krb5kdc[13467](info): commencing operation
krb5kdc: starting...
W20260808 04:39:35.391456 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.270s user 0.003s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:35 dist-test-slave-drl0 krb5kdc[13467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:39:35.539283 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39899
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:35.661564 13486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:35.661877 13486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:35.661953 13486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:35.666312 13486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:35.666435 13486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:35.666482 13486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:35.666543 13486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:35.666584 13486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:35.672353 13486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:35.673516 13486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:35.674602 13486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:35.684634 13495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:35.684849 13486 server_base.cc:1034] running on GCE node
W20260808 04:39:35.684690 13496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:35.684634 13498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:35.685601 13486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:35.687071 13486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:35.688402 13486 hybrid_clock.cc:648] HybridClock initialized: now 1786163975688347 us; error 67 us; skew 500 ppm
Aug 08 04:39:35 dist-test-slave-drl0 krb5kdc[13467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:35.692104 13486 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:39:35.693109 13486 webserver.cc:460] Webserver started at http://127.0.0.1:34911/ using document root <none> and password file <none>
I20260808 04:39:35.693661 13486 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:35.693722 13486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:35.693898 13486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:35.696440 13486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "ba94b53fd47e41cb8615a95cd6f98fe1"
format_stamp: "Formatted at 2026-08-08 04:39:35 on dist-test-slave-drl0"
server_key: "829716ff924d9385a4e49779358c36fd"
server_key_iv: "410afc0715ee3e77133957d3bf0ed164"
server_key_version: "encryptionkey@0"
I20260808 04:39:35.696977 13486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "ba94b53fd47e41cb8615a95cd6f98fe1"
format_stamp: "Formatted at 2026-08-08 04:39:35 on dist-test-slave-drl0"
server_key: "829716ff924d9385a4e49779358c36fd"
server_key_iv: "410afc0715ee3e77133957d3bf0ed164"
server_key_version: "encryptionkey@0"
I20260808 04:39:35.701576 13486 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:39:35.704226 13507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:35.705735 13486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:39:35.705909 13486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "ba94b53fd47e41cb8615a95cd6f98fe1"
format_stamp: "Formatted at 2026-08-08 04:39:35 on dist-test-slave-drl0"
server_key: "829716ff924d9385a4e49779358c36fd"
server_key_iv: "410afc0715ee3e77133957d3bf0ed164"
server_key_version: "encryptionkey@0"
I20260808 04:39:35.706032 13486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:35.711091 13486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:35.714669 13486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:35.714879 13486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:35.723067 13486 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39899
I20260808 04:39:35.723126 13575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39899 every 8 connection(s)
I20260808 04:39:35.724062 13486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:35.727619 13576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:35.728499 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 13486
I20260808 04:39:35.728662 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:35.728978 16273 external_mini_cluster.cc:1428] Setting key a8bd3cd5b867b9af8ecebd531fa61cd7
I20260808 04:39:35.734246 13576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1: Bootstrap starting.
I20260808 04:39:35.735105 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:35.730544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:58797 (local address 127.0.0.1:39899)
0808 04:39:35.731070 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:35.731087 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:35.731123 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:39:35.731883 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:35.731900 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:35.732193 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:35.732545 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:35.733376 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:35.733381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:35.733392 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:35.733403 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:35.733459 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:35.733479 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:35.733634 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:35.733706 (+ 72us) server_negotiation.cc:299] Negotiation successful
0808 04:39:35.733865 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":138,"threads_started":1}
I20260808 04:39:35.736696 13576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:35.737579 13576 log.cc:824] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:35.739187 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:39:35.740614 13576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1: No bootstrap required, opened a new log
I20260808 04:39:35.743737 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:35.741499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:43547 (local address 127.0.0.1:39899)
0808 04:39:35.741625 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:35.741628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:35.742153 (+ 525us) server_negotiation.cc:407] Connection header received
0808 04:39:35.742329 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:35.742331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:35.742363 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:35.742449 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:35.743275 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:35.743279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:35.743281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:35.743302 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:35.743327 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:35.743337 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:35.743441 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:35.743552 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 04:39:35.743595 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:39:35.746059 13576 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba94b53fd47e41cb8615a95cd6f98fe1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39899 } }
I20260808 04:39:35.746315 13576 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:35.746377 13576 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba94b53fd47e41cb8615a95cd6f98fe1, State: Initialized, Role: FOLLOWER
I20260808 04:39:35.746835 13576 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba94b53fd47e41cb8615a95cd6f98fe1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39899 } }
I20260808 04:39:35.747031 13576 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:35.747124 13576 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:35.747249 13576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:35.748232 13576 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba94b53fd47e41cb8615a95cd6f98fe1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39899 } }
I20260808 04:39:35.748596 13576 leader_election.cc:302] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba94b53fd47e41cb8615a95cd6f98fe1; no voters:
I20260808 04:39:35.748932 13576 leader_election.cc:288] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:35.749117 13588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:35.749384 13588 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1 [term 1 LEADER]: Becoming Leader. State: Replica: ba94b53fd47e41cb8615a95cd6f98fe1, State: Running, Role: LEADER
I20260808 04:39:35.749692 13588 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba94b53fd47e41cb8615a95cd6f98fe1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39899 } }
I20260808 04:39:35.750221 13576 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:35.751544 13590 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba94b53fd47e41cb8615a95cd6f98fe1. Latest consensus state: current_term: 1 leader_uuid: "ba94b53fd47e41cb8615a95cd6f98fe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba94b53fd47e41cb8615a95cd6f98fe1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39899 } } }
I20260808 04:39:35.751561 13589 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba94b53fd47e41cb8615a95cd6f98fe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba94b53fd47e41cb8615a95cd6f98fe1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39899 } } }
I20260808 04:39:35.751693 13589 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:35.751693 13590 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:35.752170 13599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:35.755125 13599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:35.760659 13599 catalog_manager.cc:1347] Generated new cluster ID: f321bf0dee554fcbb87620f167f53c8b
I20260808 04:39:35.760741 13599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:35.788568 13599 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:35.788959 13599 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:35.799187 13599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba94b53fd47e41cb8615a95cd6f98fe1: Generated new TSK 0
I20260808 04:39:35.799913 13599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:35.811388 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 13486
2026-08-08T04:39:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1717 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:35 dist-test-slave-drl0 krb5kdc[13614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:35 dist-test-slave-drl0 krb5kdc[13614](info): set up 2 sockets
Aug 08 04:39:35 dist-test-slave-drl0 krb5kdc[13614](info): commencing operation
krb5kdc: starting...
W20260808 04:39:37.075353 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.005s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:37 dist-test-slave-drl0 krb5kdc[13614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:39:37.227710 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40093
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:37.353341 13634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:37.353652 13634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:37.353725 13634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:37.357771 13634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:37.357863 13634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:37.357903 13634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:37.357959 13634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:37.357998 13634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:37.363356 13634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40093
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:37.364437 13634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:37.365481 13634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:37.375451 13644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:37.375423 13643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:37.375679 13646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:37.375871 13634 server_base.cc:1034] running on GCE node
I20260808 04:39:37.376467 13634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:37.377619 13634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:37.378839 13634 hybrid_clock.cc:648] HybridClock initialized: now 1786163977378792 us; error 63 us; skew 500 ppm
Aug 08 04:39:37 dist-test-slave-drl0 krb5kdc[13614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:37.383488 13634 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:39:37.384889 13634 webserver.cc:460] Webserver started at http://127.0.0.1:40275/ using document root <none> and password file <none>
I20260808 04:39:37.385640 13634 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:37.385761 13634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:37.386098 13634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:37.388466 13634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "f3ef12faf09f4313ad64b8b15738ad72"
format_stamp: "Formatted at 2026-08-08 04:39:37 on dist-test-slave-drl0"
server_key: "7947400321cf8bfc13b42763d3521c63"
server_key_iv: "c1e3a460fd6d407001feb204213868d0"
server_key_version: "encryptionkey@0"
I20260808 04:39:37.389050 13634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "f3ef12faf09f4313ad64b8b15738ad72"
format_stamp: "Formatted at 2026-08-08 04:39:37 on dist-test-slave-drl0"
server_key: "7947400321cf8bfc13b42763d3521c63"
server_key_iv: "c1e3a460fd6d407001feb204213868d0"
server_key_version: "encryptionkey@0"
I20260808 04:39:37.393164 13634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260808 04:39:37.395936 13658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:37.397637 13634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 04:39:37.397815 13634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "f3ef12faf09f4313ad64b8b15738ad72"
format_stamp: "Formatted at 2026-08-08 04:39:37 on dist-test-slave-drl0"
server_key: "7947400321cf8bfc13b42763d3521c63"
server_key_iv: "c1e3a460fd6d407001feb204213868d0"
server_key_version: "encryptionkey@0"
I20260808 04:39:37.397940 13634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:37.419579 13634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:37.423825 13634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:37.424068 13634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:37.432226 13634 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40093
I20260808 04:39:37.432243 13725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40093 every 8 connection(s)
I20260808 04:39:37.433245 13634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:37.436118 13727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:37.436690 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 13634
I20260808 04:39:37.436856 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:37.437186 16273 external_mini_cluster.cc:1428] Setting key 536d6a290be5a1d6399e0d49f9783649
I20260808 04:39:37.442970 13727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72: Bootstrap starting.
I20260808 04:39:37.445741 13727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:37.446758 13727 log.cc:824] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:39:37 dist-test-slave-drl0 krb5kdc[13614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 04:39:37.449244 13727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72: No bootstrap required, opened a new log
I20260808 04:39:37.453508 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:37.438838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:48683 (local address 127.0.0.1:40093)
0808 04:39:37.439330 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:37.439345 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:37.439382 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:39:37.440063 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:37.440079 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:37.440413 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:37.440692 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:37.441695 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:37.442745 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:37.443985 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:37.444616 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:37.444748 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:37.448293 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:37.448300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:37.448314 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:37.448342 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:37.450115 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:37.450779 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:37.450793 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:37.450798 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:37.450992 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:37.451402 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:37.451405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:37.451406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:37.451809 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:37.451992 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:37.452436 (+ 444us) server_negotiation.cc:299] Negotiation successful
0808 04:39:37.452744 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":151,"threads_started":1}
I20260808 04:39:37.454048 13727 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ef12faf09f4313ad64b8b15738ad72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40093 } }
I20260808 04:39:37.454352 13727 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:37.454418 13727 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3ef12faf09f4313ad64b8b15738ad72, State: Initialized, Role: FOLLOWER
I20260808 04:39:37.454989 13727 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ef12faf09f4313ad64b8b15738ad72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40093 } }
I20260808 04:39:37.455250 13727 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:37.455372 13727 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:37.455511 13727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:37.456728 13727 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ef12faf09f4313ad64b8b15738ad72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40093 } }
I20260808 04:39:37.457096 13727 leader_election.cc:302] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3ef12faf09f4313ad64b8b15738ad72; no voters:
I20260808 04:39:37.457469 13727 leader_election.cc:288] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:37.457777 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:39:37.457744 13734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:37.458173 13734 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72 [term 1 LEADER]: Becoming Leader. State: Replica: f3ef12faf09f4313ad64b8b15738ad72, State: Running, Role: LEADER
I20260808 04:39:37.458592 13734 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ef12faf09f4313ad64b8b15738ad72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40093 } }
I20260808 04:39:37.459064 13727 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:37.461140 13735 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3ef12faf09f4313ad64b8b15738ad72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ef12faf09f4313ad64b8b15738ad72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40093 } } }
I20260808 04:39:37.461349 13735 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:37.461860 13757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:37.461920 13736 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3ef12faf09f4313ad64b8b15738ad72. Latest consensus state: current_term: 1 leader_uuid: "f3ef12faf09f4313ad64b8b15738ad72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3ef12faf09f4313ad64b8b15738ad72" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40093 } } }
I20260808 04:39:37.462146 13736 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:37.465368 13757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:37.470477 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:37.460383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:34987 (local address 127.0.0.1:40093)
0808 04:39:37.461741 (+ 1358us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:37.461747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:37.462381 (+ 634us) server_negotiation.cc:407] Connection header received
0808 04:39:37.462528 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:37.462534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:37.462594 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:37.462675 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:37.463460 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:37.464052 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:37.464874 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:37.465107 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:37.465220 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:37.466663 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:37.466667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:37.466671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:37.466700 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:37.468388 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:37.469041 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:37.469044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:37.469046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:37.469133 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:37.469518 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:37.469522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:37.469523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:37.469744 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:37.469909 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:37.470204 (+ 295us) server_negotiation.cc:299] Negotiation successful
0808 04:39:37.470312 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1248}
I20260808 04:39:37.473456 13757 catalog_manager.cc:1347] Generated new cluster ID: 86d36622ff7d429a934c42a1dcb88f66
I20260808 04:39:37.473568 13757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:37.483009 13757 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:37.484349 13757 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:37.496352 13757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3ef12faf09f4313ad64b8b15738ad72: Generated new TSK 0
I20260808 04:39:37.497587 13757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:37.505131 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 13634
W20260808 04:39:37.518676 13610 connection.cc:535] client connection to 127.0.0.1:40093 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40093): Connection reset by peer (error 104)
2026-08-08T04:39:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1694 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:37 dist-test-slave-drl0 krb5kdc[13769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:37 dist-test-slave-drl0 krb5kdc[13769](info): set up 2 sockets
Aug 08 04:39:37 dist-test-slave-drl0 krb5kdc[13769](info): commencing operation
krb5kdc: starting...
W20260808 04:39:38.796222 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:38 dist-test-slave-drl0 krb5kdc[13769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:39:38.951786 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:39.083278 13791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:39.083636 13791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:39.083716 13791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:39.088171 13791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:39.088294 13791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:39.088338 13791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:39.088395 13791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:39.088433 13791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:39.095151 13791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46831
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:39.096618 13791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:39.098544 13791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:39.108989 13797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:39.109191 13800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:39.109256 13802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:39.109560 13791 server_base.cc:1034] running on GCE node
I20260808 04:39:39.110239 13791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:39.111465 13791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:39.112651 13791 hybrid_clock.cc:648] HybridClock initialized: now 1786163979112615 us; error 51 us; skew 500 ppm
Aug 08 04:39:39 dist-test-slave-drl0 krb5kdc[13769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:39.119174 13791 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:39:39.120195 13791 webserver.cc:460] Webserver started at http://127.0.0.1:37495/ using document root <none> and password file <none>
I20260808 04:39:39.121222 13791 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:39.121371 13791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:39.121757 13791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:39.124853 13791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "1ed11fe3a4ef4bc9afa8ba487ef6985d"
format_stamp: "Formatted at 2026-08-08 04:39:39 on dist-test-slave-drl0"
server_key: "6492d777fe6d72ffa1d1b86d7d1d269f"
server_key_iv: "e912b0a10792cd7c6d44d40aee29bd4d"
server_key_version: "encryptionkey@0"
I20260808 04:39:39.125600 13791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "1ed11fe3a4ef4bc9afa8ba487ef6985d"
format_stamp: "Formatted at 2026-08-08 04:39:39 on dist-test-slave-drl0"
server_key: "6492d777fe6d72ffa1d1b86d7d1d269f"
server_key_iv: "e912b0a10792cd7c6d44d40aee29bd4d"
server_key_version: "encryptionkey@0"
I20260808 04:39:39.130178 13791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 04:39:39.133553 13812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:39.135174 13791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:39:39.135350 13791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "1ed11fe3a4ef4bc9afa8ba487ef6985d"
format_stamp: "Formatted at 2026-08-08 04:39:39 on dist-test-slave-drl0"
server_key: "6492d777fe6d72ffa1d1b86d7d1d269f"
server_key_iv: "e912b0a10792cd7c6d44d40aee29bd4d"
server_key_version: "encryptionkey@0"
I20260808 04:39:39.135480 13791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:39.148661 13791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:39.152428 13791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:39.152704 13791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:39.162550 13791 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46831
I20260808 04:39:39.162698 13882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46831 every 8 connection(s)
I20260808 04:39:39.163971 13791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:39.167411 13883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:39.171806 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 13791
I20260808 04:39:39.172106 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:39.172493 16273 external_mini_cluster.cc:1428] Setting key 4eb8fd5dd44758d58bfb924757370cb5
I20260808 04:39:39.175408 13883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d: Bootstrap starting.
I20260808 04:39:39.178539 13883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:39.179536 13883 log.cc:824] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:39.182679 13883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d: No bootstrap required, opened a new log
I20260808 04:39:39.185981 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:39.174348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:33597 (local address 127.0.0.1:46831)
0808 04:39:39.174971 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:39.174995 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:39.175047 (+ 52us) server_negotiation.cc:407] Connection header received
0808 04:39:39.176019 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:39.176045 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:39.176514 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:39.176953 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:39.177420 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:39.179122 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:39.180594 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:39.181596 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:39.181787 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:39.183136 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:39.183144 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:39.183166 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:39.183758 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:39.183841 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:39.183872 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:39.184059 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:39.184267 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 04:39:39.184466 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":124,"threads_started":1}
I20260808 04:39:39.186590 13883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed11fe3a4ef4bc9afa8ba487ef6985d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46831 } }
I20260808 04:39:39.186952 13883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:39.187013 13883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed11fe3a4ef4bc9afa8ba487ef6985d, State: Initialized, Role: FOLLOWER
I20260808 04:39:39.187527 13883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed11fe3a4ef4bc9afa8ba487ef6985d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46831 } }
I20260808 04:39:39.187707 13883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:39.187805 13883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:39.187948 13883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:39.189272 13883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed11fe3a4ef4bc9afa8ba487ef6985d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46831 } }
I20260808 04:39:39.189677 13883 leader_election.cc:302] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ed11fe3a4ef4bc9afa8ba487ef6985d; no voters:
I20260808 04:39:39.190135 13883 leader_election.cc:288] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:39.190208 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:39:39.190282 13891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:39.190550 13891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d [term 1 LEADER]: Becoming Leader. State: Replica: 1ed11fe3a4ef4bc9afa8ba487ef6985d, State: Running, Role: LEADER
I20260808 04:39:39.190886 13891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed11fe3a4ef4bc9afa8ba487ef6985d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46831 } }
I20260808 04:39:39.191893 13883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:39.192060 13891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ed11fe3a4ef4bc9afa8ba487ef6985d. Latest consensus state: current_term: 1 leader_uuid: "1ed11fe3a4ef4bc9afa8ba487ef6985d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed11fe3a4ef4bc9afa8ba487ef6985d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46831 } } }
I20260808 04:39:39.192188 13891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:39.192755 13892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ed11fe3a4ef4bc9afa8ba487ef6985d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed11fe3a4ef4bc9afa8ba487ef6985d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46831 } } }
I20260808 04:39:39.192934 13892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:39.193403 13903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:39.196640 13903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:39.204422 13903 catalog_manager.cc:1347] Generated new cluster ID: 2da3ea714b914ed397f760968005dbda
I20260808 04:39:39.204546 13903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:39.204824 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:39.193018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:52189 (local address 127.0.0.1:46831)
0808 04:39:39.193139 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:39.193144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:39.198715 (+ 5571us) server_negotiation.cc:407] Connection header received
0808 04:39:39.198875 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:39.198878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:39.198986 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:39.199093 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:39.199592 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:39.200554 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:39.202117 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:39.202512 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:39.202661 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:39.204208 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:39.204213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:39.204217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:39.204233 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:39.204274 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:39.204290 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:39.204396 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:39.204603 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 04:39:39.204648 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 04:39:39.228964 13903 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:39.230341 13903 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:39.237319 13903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ed11fe3a4ef4bc9afa8ba487ef6985d: Generated new TSK 0
I20260808 04:39:39.238222 13903 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:39.250538 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 13791
W20260808 04:39:39.266927 13764 connection.cc:535] client connection to 127.0.0.1:46831 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46831): Connection reset by peer (error 104)
2026-08-08T04:39:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1748 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:39 dist-test-slave-drl0 krb5kdc[13925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:39 dist-test-slave-drl0 krb5kdc[13925](info): set up 2 sockets
Aug 08 04:39:39 dist-test-slave-drl0 krb5kdc[13925](info): commencing operation
krb5kdc: starting...
W20260808 04:39:40.553514 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.270s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:40 dist-test-slave-drl0 krb5kdc[13925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 04:39:40.712297 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36651
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:40.846805 13952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:40.847195 13952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:40.847295 13952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:40.852066 13952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:40.852178 13952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:40.852216 13952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:40.852257 13952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:40.852293 13952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:40.861559 13952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36651
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:40.863350 13952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:40.865061 13952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:40.875919 13960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:40.876080 13952 server_base.cc:1034] running on GCE node
W20260808 04:39:40.875938 13964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:40.875938 13962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:40.876925 13952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:40.878207 13952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:40.879451 13952 hybrid_clock.cc:648] HybridClock initialized: now 1786163980879394 us; error 70 us; skew 500 ppm
Aug 08 04:39:40 dist-test-slave-drl0 krb5kdc[13925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:40.883352 13952 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 04:39:40.884531 13952 webserver.cc:460] Webserver started at http://127.0.0.1:39167/ using document root <none> and password file <none>
I20260808 04:39:40.885170 13952 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:40.885269 13952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:40.885526 13952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:40.887792 13952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "363fa512b397437f816c79a0d9344943"
format_stamp: "Formatted at 2026-08-08 04:39:40 on dist-test-slave-drl0"
server_key: "25bd3d4d9ee5a72ad9f736000820f8e6"
server_key_iv: "7efd49823d6d6e01d880853fea5e6d8b"
server_key_version: "encryptionkey@0"
I20260808 04:39:40.888401 13952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "363fa512b397437f816c79a0d9344943"
format_stamp: "Formatted at 2026-08-08 04:39:40 on dist-test-slave-drl0"
server_key: "25bd3d4d9ee5a72ad9f736000820f8e6"
server_key_iv: "7efd49823d6d6e01d880853fea5e6d8b"
server_key_version: "encryptionkey@0"
I20260808 04:39:40.892449 13952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:39:40.895601 13974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:40.897680 13952 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:39:40.897945 13952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "363fa512b397437f816c79a0d9344943"
format_stamp: "Formatted at 2026-08-08 04:39:40 on dist-test-slave-drl0"
server_key: "25bd3d4d9ee5a72ad9f736000820f8e6"
server_key_iv: "7efd49823d6d6e01d880853fea5e6d8b"
server_key_version: "encryptionkey@0"
I20260808 04:39:40.898224 13952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:40.903321 13952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:40.907991 13952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:40.908299 13952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:40.919373 13952 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36651
I20260808 04:39:40.919451 14044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36651 every 8 connection(s)
I20260808 04:39:40.920514 13952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:40.922465 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 13952
I20260808 04:39:40.922675 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:40.923154 16273 external_mini_cluster.cc:1428] Setting key 0f971767b4cf8d00f3dd1c2a220ad2cc
I20260808 04:39:40.925333 14045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:40.930364 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:40.925165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:33257 (local address 127.0.0.1:36651)
0808 04:39:40.925733 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:40.925750 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:40.925791 (+ 41us) server_negotiation.cc:407] Connection header received
0808 04:39:40.926486 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:40.926504 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:40.926793 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:40.927464 (+ 671us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:40.928531 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:40.928536 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:40.928566 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:40.928584 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:40.928676 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:40.928707 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:40.928896 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:40.928979 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 04:39:40.929154 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":193,"threads_started":1}
I20260808 04:39:40.934175 14045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943: Bootstrap starting.
I20260808 04:39:40.934476 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:39:40.937160 14045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:40.938201 14045 log.cc:824] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:40.940310 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:40.937462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:41901 (local address 127.0.0.1:36651)
0808 04:39:40.937903 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:40.937909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:40.938197 (+ 288us) server_negotiation.cc:407] Connection header received
0808 04:39:40.938431 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:40.938435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:40.938482 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:40.938548 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:40.939563 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:40.939567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:40.939569 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:40.939582 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:40.939610 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:40.939624 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:40.939958 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:40.940026 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 04:39:40.940100 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331}
I20260808 04:39:40.941524 14045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943: No bootstrap required, opened a new log
I20260808 04:39:40.946449 14045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363fa512b397437f816c79a0d9344943" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36651 } }
I20260808 04:39:40.946779 14045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:40.946892 14045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363fa512b397437f816c79a0d9344943, State: Initialized, Role: FOLLOWER
I20260808 04:39:40.947408 14045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363fa512b397437f816c79a0d9344943" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36651 } }
I20260808 04:39:40.947578 14045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:40.947671 14045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:40.947796 14045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:40.948853 14045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363fa512b397437f816c79a0d9344943" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36651 } }
I20260808 04:39:40.949229 14045 leader_election.cc:302] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 363fa512b397437f816c79a0d9344943; no voters:
I20260808 04:39:40.949581 14045 leader_election.cc:288] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:40.949784 14065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:40.950116 14065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943 [term 1 LEADER]: Becoming Leader. State: Replica: 363fa512b397437f816c79a0d9344943, State: Running, Role: LEADER
I20260808 04:39:40.950488 14065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363fa512b397437f816c79a0d9344943" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36651 } }
I20260808 04:39:40.951014 14045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:40.952297 14067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 363fa512b397437f816c79a0d9344943. Latest consensus state: current_term: 1 leader_uuid: "363fa512b397437f816c79a0d9344943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363fa512b397437f816c79a0d9344943" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36651 } } }
I20260808 04:39:40.952435 14067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:40.953440 14066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "363fa512b397437f816c79a0d9344943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "363fa512b397437f816c79a0d9344943" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36651 } } }
I20260808 04:39:40.953835 14066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:40.953634 14079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:40.957505 14079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:40.965219 14079 catalog_manager.cc:1347] Generated new cluster ID: 2d2933f60690439c90bac5b09c54d3a8
I20260808 04:39:40.965317 14079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:40.976263 14079 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:40.976603 14079 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:40.985780 14079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 363fa512b397437f816c79a0d9344943: Generated new TSK 0
I20260808 04:39:40.986615 14079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:40.997411 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 13952
2026-08-08T04:39:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1743 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:41 dist-test-slave-drl0 krb5kdc[14094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:41 dist-test-slave-drl0 krb5kdc[14094](info): set up 2 sockets
Aug 08 04:39:41 dist-test-slave-drl0 krb5kdc[14094](info): commencing operation
krb5kdc: starting...
W20260808 04:39:42.308845 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.283s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:42 dist-test-slave-drl0 krb5kdc[14094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:39:42.464253 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:38801
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:45955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:38801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:42.596645 14112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:42.597080 14112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:42.597178 14112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:42.601884 14112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:42.602281 14112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:42.602371 14112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:42.602417 14112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:42.602458 14112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:42.609973 14112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:45955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:38801
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:38801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:42.611404 14112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:42.612619 14112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:42.624219 14128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:42.624303 14112 server_base.cc:1034] running on GCE node
W20260808 04:39:42.624238 14123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:42.624238 14121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:42.625497 14112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:42.627381 14112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:42.628610 14112 hybrid_clock.cc:648] HybridClock initialized: now 1786163982628573 us; error 51 us; skew 500 ppm
Aug 08 04:39:42 dist-test-slave-drl0 krb5kdc[14094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163982, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:42.632160 14112 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:39:42.633196 14112 webserver.cc:460] Webserver started at http://127.15.228.126:43003/ using document root <none> and password file <none>
I20260808 04:39:42.633805 14112 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:42.633901 14112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:42.634147 14112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:42.636417 14112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "5470f493db4a41b18ad744090a2e4f22"
format_stamp: "Formatted at 2026-08-08 04:39:42 on dist-test-slave-drl0"
server_key: "ffae9713fd0cf09082e30b33681d0f8d"
server_key_iv: "1a3a60b877467f883d91eb19b94b2e71"
server_key_version: "encryptionkey@0"
I20260808 04:39:42.636999 14112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "5470f493db4a41b18ad744090a2e4f22"
format_stamp: "Formatted at 2026-08-08 04:39:42 on dist-test-slave-drl0"
server_key: "ffae9713fd0cf09082e30b33681d0f8d"
server_key_iv: "1a3a60b877467f883d91eb19b94b2e71"
server_key_version: "encryptionkey@0"
I20260808 04:39:42.641605 14112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 04:39:42.644491 14140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:42.646467 14112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:39:42.646749 14112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "5470f493db4a41b18ad744090a2e4f22"
format_stamp: "Formatted at 2026-08-08 04:39:42 on dist-test-slave-drl0"
server_key: "ffae9713fd0cf09082e30b33681d0f8d"
server_key_iv: "1a3a60b877467f883d91eb19b94b2e71"
server_key_version: "encryptionkey@0"
I20260808 04:39:42.646932 14112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:42.664438 14112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:42.669627 14112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:42.669889 14112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:42.678917 14112 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:38801
I20260808 04:39:42.679080 14215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:38801 every 8 connection(s)
I20260808 04:39:42.680145 14112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:42.684419 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 14112
I20260808 04:39:42.684726 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:42.684725 14216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:42.685142 16273 external_mini_cluster.cc:1428] Setting key d584bd39d726dabaa8c92119423725a7
I20260808 04:39:42.693269 14216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22: Bootstrap starting.
Aug 08 04:39:42 dist-test-slave-drl0 krb5kdc[14094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:39:42.696589 14216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:42.698037 14216 log.cc:824] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:42.701848 14216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22: No bootstrap required, opened a new log
I20260808 04:39:42.704193 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:42.686736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39557 (local address 127.15.228.126:38801)
0808 04:39:42.687257 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:42.687283 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:42.687327 (+ 44us) server_negotiation.cc:407] Connection header received
0808 04:39:42.688053 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:42.688076 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:42.688532 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:42.688907 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:42.689650 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:42.691199 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:42.692276 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:42.693079 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:42.693307 (+ 228us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:42.696421 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:42.696431 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:42.696451 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:42.696492 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:42.699107 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:42.700098 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:42.700109 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:42.700118 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:42.700342 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:42.700999 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:42.701003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:42.701004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:42.701731 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:42.701997 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:42.702527 (+ 530us) server_negotiation.cc:299] Negotiation successful
0808 04:39:42.702786 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":195,"threads_started":1}
I20260808 04:39:42.707082 14216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5470f493db4a41b18ad744090a2e4f22" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 38801 } }
I20260808 04:39:42.707464 14216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:42.707530 14216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5470f493db4a41b18ad744090a2e4f22, State: Initialized, Role: FOLLOWER
I20260808 04:39:42.708271 14216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5470f493db4a41b18ad744090a2e4f22" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 38801 } }
I20260808 04:39:42.708459 14216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:42.708559 14216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:42.708671 14216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:42.709271 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:39:42.710100 14216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5470f493db4a41b18ad744090a2e4f22" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 38801 } }
I20260808 04:39:42.710520 14216 leader_election.cc:302] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5470f493db4a41b18ad744090a2e4f22; no voters:
I20260808 04:39:42.710919 14216 leader_election.cc:288] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:42.711082 14227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:42.711369 14227 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22 [term 1 LEADER]: Becoming Leader. State: Replica: 5470f493db4a41b18ad744090a2e4f22, State: Running, Role: LEADER
I20260808 04:39:42.712219 14227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5470f493db4a41b18ad744090a2e4f22" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 38801 } }
I20260808 04:39:42.712373 14216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:42.714730 14231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5470f493db4a41b18ad744090a2e4f22. Latest consensus state: current_term: 1 leader_uuid: "5470f493db4a41b18ad744090a2e4f22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5470f493db4a41b18ad744090a2e4f22" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 38801 } } }
I20260808 04:39:42.714915 14230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5470f493db4a41b18ad744090a2e4f22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5470f493db4a41b18ad744090a2e4f22" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 38801 } } }
I20260808 04:39:42.715101 14230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:42.715101 14231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22 [sys.catalog]: This master's current role is: LEADER
W20260808 04:39:42.718057 14246 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:39:42.718150 14246 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:39:42.718238 14248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:42.719278 14248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:42.724572 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:42.711779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45305 (local address 127.15.228.126:38801)
0808 04:39:42.711913 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:42.711918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:42.715526 (+ 3608us) server_negotiation.cc:407] Connection header received
0808 04:39:42.715759 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:42.715763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:42.715812 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:42.715867 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:42.716777 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:42.717310 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:42.718141 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:42.718483 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:42.718600 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:42.720360 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:42.720364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:42.720367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:42.720393 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:42.722232 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:42.722850 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:42.722854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:42.722856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:42.722975 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:42.723360 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:42.723364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:42.723366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:42.723598 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:42.723754 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:42.724303 (+ 549us) server_negotiation.cc:299] Negotiation successful
0808 04:39:42.724408 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 04:39:42.731348 14248 catalog_manager.cc:1347] Generated new cluster ID: 7877d9ca07b54992b1a4267e8f3101d9
I20260808 04:39:42.731428 14248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:42.746307 14248 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:42.748185 14248 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:42.756871 14248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5470f493db4a41b18ad744090a2e4f22: Generated new TSK 0
I20260808 04:39:42.757704 14248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:42.764518 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 14112
W20260808 04:39:42.780906 14089 connection.cc:535] client connection to 127.15.228.126:38801 recv error: Network error: failed to read from TLS socket (remote: 127.15.228.126:38801): Connection reset by peer (error 104)
2026-08-08T04:39:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1769 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:42 dist-test-slave-drl0 krb5kdc[14255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:42 dist-test-slave-drl0 krb5kdc[14255](info): set up 2 sockets
Aug 08 04:39:42 dist-test-slave-drl0 krb5kdc[14255](info): commencing operation
krb5kdc: starting...
W20260808 04:39:44.062232 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.266s user 0.006s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:44 dist-test-slave-drl0 krb5kdc[14255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:39:44.208681 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:36351
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:44051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:36351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:44.340121 14276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:44.340399 14276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:44.340442 14276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:44.344707 14276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:44.344781 14276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:44.344802 14276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:44.344820 14276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:44.344835 14276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:44.350340 14276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:44051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:36351
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:36351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:44.351485 14276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:44.352535 14276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:44.361726 14283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:44.361920 14282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:44.362020 14276 server_base.cc:1034] running on GCE node
W20260808 04:39:44.361920 14285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:44.362910 14276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:44.364605 14276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:44.365866 14276 hybrid_clock.cc:648] HybridClock initialized: now 1786163984365791 us; error 81 us; skew 500 ppm
Aug 08 04:39:44 dist-test-slave-drl0 krb5kdc[14255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163984, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:44.373869 14276 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:39:44.375329 14276 webserver.cc:460] Webserver started at http://127.15.228.126:42455/ using document root <none> and password file <none>
I20260808 04:39:44.376088 14276 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:44.376196 14276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:44.376492 14276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:44.379210 14276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "e0f9a094cebf42b6b6b532b95e083bdd"
format_stamp: "Formatted at 2026-08-08 04:39:44 on dist-test-slave-drl0"
server_key: "c8c5cbc3e7d39b542daf8026a6ca6514"
server_key_iv: "9e836bcac4972a278ad4d4e2f342dd75"
server_key_version: "encryptionkey@0"
I20260808 04:39:44.379940 14276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "e0f9a094cebf42b6b6b532b95e083bdd"
format_stamp: "Formatted at 2026-08-08 04:39:44 on dist-test-slave-drl0"
server_key: "c8c5cbc3e7d39b542daf8026a6ca6514"
server_key_iv: "9e836bcac4972a278ad4d4e2f342dd75"
server_key_version: "encryptionkey@0"
I20260808 04:39:44.385905 14276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 04:39:44.389645 14294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:44.391426 14276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 04:39:44.391626 14276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "e0f9a094cebf42b6b6b532b95e083bdd"
format_stamp: "Formatted at 2026-08-08 04:39:44 on dist-test-slave-drl0"
server_key: "c8c5cbc3e7d39b542daf8026a6ca6514"
server_key_iv: "9e836bcac4972a278ad4d4e2f342dd75"
server_key_version: "encryptionkey@0"
I20260808 04:39:44.391793 14276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:44.414460 14276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:44.418807 14276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:44.419189 14276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:44.429196 14276 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:36351
I20260808 04:39:44.429261 14367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:36351 every 8 connection(s)
I20260808 04:39:44.430290 14276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:44.434242 14368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:44.437455 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 14276
I20260808 04:39:44.437592 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:44.437901 16273 external_mini_cluster.cc:1428] Setting key e2efe1e9cdf9b17e0785aa0c8ce04f3e
I20260808 04:39:44.441927 14368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd: Bootstrap starting.
I20260808 04:39:44.444761 14368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:44.445634 14368 log.cc:824] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:44.448032 14368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd: No bootstrap required, opened a new log
I20260808 04:39:44.449239 14371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:44.439921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54163 (local address 127.15.228.126:36351)
0808 04:39:44.440502 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:44.440523 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:44.440569 (+ 46us) server_negotiation.cc:407] Connection header received
0808 04:39:44.441512 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:44.441538 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:44.441929 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:44.442296 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:44.442684 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:44.443769 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:44.445031 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:44.445508 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:44.445639 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:44.446656 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:44.446662 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:44.446674 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:44.447431 (+ 757us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:44.447551 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:44.447590 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:44.447824 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:44.448085 (+ 261us) server_negotiation.cc:299] Negotiation successful
0808 04:39:44.448282 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":178,"threads_started":1}
I20260808 04:39:44.452646 14368 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f9a094cebf42b6b6b532b95e083bdd" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36351 } }
I20260808 04:39:44.452947 14368 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:44.453001 14368 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f9a094cebf42b6b6b532b95e083bdd, State: Initialized, Role: FOLLOWER
I20260808 04:39:44.453459 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:39:44.453482 14368 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f9a094cebf42b6b6b532b95e083bdd" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36351 } }
I20260808 04:39:44.453670 14368 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:44.453737 14368 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:44.453838 14368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:44.454888 14368 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f9a094cebf42b6b6b532b95e083bdd" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36351 } }
I20260808 04:39:44.455268 14368 leader_election.cc:302] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0f9a094cebf42b6b6b532b95e083bdd; no voters:
I20260808 04:39:44.455626 14368 leader_election.cc:288] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:44.455711 14383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:44.455943 14383 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd [term 1 LEADER]: Becoming Leader. State: Replica: e0f9a094cebf42b6b6b532b95e083bdd, State: Running, Role: LEADER
I20260808 04:39:44.456269 14383 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f9a094cebf42b6b6b532b95e083bdd" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36351 } }
I20260808 04:39:44.457026 14368 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:44.457937 14389 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0f9a094cebf42b6b6b532b95e083bdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f9a094cebf42b6b6b532b95e083bdd" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36351 } } }
I20260808 04:39:44.458099 14389 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:44.458554 14386 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0f9a094cebf42b6b6b532b95e083bdd. Latest consensus state: current_term: 1 leader_uuid: "e0f9a094cebf42b6b6b532b95e083bdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f9a094cebf42b6b6b532b95e083bdd" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36351 } } }
I20260808 04:39:44.458706 14386 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:44.459100 14398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:44.461804 14371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:44.455781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47045 (local address 127.15.228.126:36351)
0808 04:39:44.455906 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:44.455910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:44.456418 (+ 508us) server_negotiation.cc:407] Connection header received
0808 04:39:44.456607 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:44.456613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:44.456669 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:44.456745 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:44.457117 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:44.457876 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:44.459418 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:44.459798 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:44.459913 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:44.461176 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:44.461180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:44.461184 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:44.461206 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:44.461246 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:44.461260 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:44.461371 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:44.461579 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 04:39:44.461632 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 04:39:44.463399 14398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:44.470278 14398 catalog_manager.cc:1347] Generated new cluster ID: 3cd57c44b4e6425b944e0749876d46f2
I20260808 04:39:44.470364 14398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:44.515625 14398 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:44.516657 14398 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:44.521404 14398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0f9a094cebf42b6b6b532b95e083bdd: Generated new TSK 0
I20260808 04:39:44.522439 14398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:44.535547 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 14276
W20260808 04:39:44.549860 14253 connection.cc:535] client connection to 127.15.228.126:36351 recv error: Network error: failed to read from TLS socket (remote: 127.15.228.126:36351): Connection reset by peer (error 104)
2026-08-08T04:39:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1768 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:44 dist-test-slave-drl0 krb5kdc[14417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:44 dist-test-slave-drl0 krb5kdc[14417](info): set up 2 sockets
Aug 08 04:39:44 dist-test-slave-drl0 krb5kdc[14417](info): commencing operation
krb5kdc: starting...
W20260808 04:39:45.855819 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.290s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:45 dist-test-slave-drl0 krb5kdc[14417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:39:46.011965 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:34825
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:46473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:34825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:46.140009 14436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:46.140287 14436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:46.140329 14436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:46.145232 14436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:46.145321 14436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:46.145344 14436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:46.145361 14436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:46.145378 14436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:46.151113 14436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:46473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:34825
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:34825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:46.152895 14436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:46.154074 14436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:46.163691 14441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:46.163661 14442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:46.163666 14446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:46.164330 14436 server_base.cc:1034] running on GCE node
I20260808 04:39:46.165105 14436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:46.166599 14436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:46.167937 14436 hybrid_clock.cc:648] HybridClock initialized: now 1786163986167883 us; error 78 us; skew 500 ppm
Aug 08 04:39:46 dist-test-slave-drl0 krb5kdc[14417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163986, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:46.171425 14436 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:39:46.172505 14436 webserver.cc:460] Webserver started at http://127.15.228.126:44809/ using document root <none> and password file <none>
I20260808 04:39:46.173102 14436 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:46.173167 14436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:46.173355 14436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:46.175746 14436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "80922fb804d44faba5e2131fc4d6db26"
format_stamp: "Formatted at 2026-08-08 04:39:46 on dist-test-slave-drl0"
server_key: "c12fc1a7d99aae7e17a505262cd5bca6"
server_key_iv: "3a92936e994b0438cc84f358c7ffbc13"
server_key_version: "encryptionkey@0"
I20260808 04:39:46.176294 14436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "80922fb804d44faba5e2131fc4d6db26"
format_stamp: "Formatted at 2026-08-08 04:39:46 on dist-test-slave-drl0"
server_key: "c12fc1a7d99aae7e17a505262cd5bca6"
server_key_iv: "3a92936e994b0438cc84f358c7ffbc13"
server_key_version: "encryptionkey@0"
I20260808 04:39:46.180707 14436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 04:39:46.183532 14455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:46.185369 14436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:39:46.185572 14436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "80922fb804d44faba5e2131fc4d6db26"
format_stamp: "Formatted at 2026-08-08 04:39:46 on dist-test-slave-drl0"
server_key: "c12fc1a7d99aae7e17a505262cd5bca6"
server_key_iv: "3a92936e994b0438cc84f358c7ffbc13"
server_key_version: "encryptionkey@0"
I20260808 04:39:46.185712 14436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:46.191766 14436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:46.202065 14436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:46.202345 14436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:46.210433 14527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:34825 every 8 connection(s)
I20260808 04:39:46.210448 14436 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:34825
I20260808 04:39:46.211725 14436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:46.215520 14528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:46.221097 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 14436
I20260808 04:39:46.221249 14528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26: Bootstrap starting.
I20260808 04:39:46.221274 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:46.221572 16273 external_mini_cluster.cc:1428] Setting key eb05eb8df3b084543d8f2f0c06ff968c
I20260808 04:39:46.224038 14528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:46.224956 14528 log.cc:824] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:46.227463 14528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26: No bootstrap required, opened a new log
I20260808 04:39:46.227612 14533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:46.223054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42773 (local address 127.15.228.126:34825)
0808 04:39:46.223504 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:46.223523 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:46.223564 (+ 41us) server_negotiation.cc:407] Connection header received
0808 04:39:46.224507 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:46.224531 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:46.224898 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:46.225277 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:46.226123 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:46.226129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:46.226143 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:46.226156 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:46.226222 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:46.226247 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:46.226360 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:46.226530 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 04:39:46.226683 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":110,"threads_started":1}
I20260808 04:39:46.230501 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:39:46.231637 14528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80922fb804d44faba5e2131fc4d6db26" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34825 } }
I20260808 04:39:46.232088 14528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:46.232172 14528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80922fb804d44faba5e2131fc4d6db26, State: Initialized, Role: FOLLOWER
I20260808 04:39:46.232718 14528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80922fb804d44faba5e2131fc4d6db26" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34825 } }
I20260808 04:39:46.232926 14528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:46.233021 14528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:46.233121 14528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:46.234283 14528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80922fb804d44faba5e2131fc4d6db26" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34825 } }
I20260808 04:39:46.234655 14528 leader_election.cc:302] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80922fb804d44faba5e2131fc4d6db26; no voters:
I20260808 04:39:46.235040 14528 leader_election.cc:288] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:46.235111 14543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:46.235364 14543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26 [term 1 LEADER]: Becoming Leader. State: Replica: 80922fb804d44faba5e2131fc4d6db26, State: Running, Role: LEADER
I20260808 04:39:46.235745 14543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80922fb804d44faba5e2131fc4d6db26" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34825 } }
I20260808 04:39:46.236089 14533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:46.233279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51643 (local address 127.15.228.126:34825)
0808 04:39:46.233441 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:46.233448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:46.233887 (+ 439us) server_negotiation.cc:407] Connection header received
0808 04:39:46.234009 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:46.234012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:46.234052 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:46.234160 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:46.235440 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:46.235445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:46.235449 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:46.235466 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:46.235516 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:46.235529 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:46.235799 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:46.235885 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 04:39:46.235929 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 04:39:46.236482 14528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:46.237061 14543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80922fb804d44faba5e2131fc4d6db26. Latest consensus state: current_term: 1 leader_uuid: "80922fb804d44faba5e2131fc4d6db26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80922fb804d44faba5e2131fc4d6db26" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34825 } } }
I20260808 04:39:46.237215 14543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:46.237457 14543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80922fb804d44faba5e2131fc4d6db26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80922fb804d44faba5e2131fc4d6db26" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 34825 } } }
I20260808 04:39:46.237604 14548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:46.237620 14543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:46.240968 14548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:46.247996 14548 catalog_manager.cc:1347] Generated new cluster ID: 75e801e58e1f47a98d7b74a6b6865b98
I20260808 04:39:46.248093 14548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:46.279433 14548 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:46.279796 14548 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:46.285946 14548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80922fb804d44faba5e2131fc4d6db26: Generated new TSK 0
I20260808 04:39:46.286880 14548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:46.303543 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 14436
2026-08-08T04:39:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1767 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:46 dist-test-slave-drl0 krb5kdc[14573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:46 dist-test-slave-drl0 krb5kdc[14573](info): set up 2 sockets
Aug 08 04:39:46 dist-test-slave-drl0 krb5kdc[14573](info): commencing operation
krb5kdc: starting...
W20260808 04:39:47.624689 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.292s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:47 dist-test-slave-drl0 krb5kdc[14573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:39:47.782703 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:41795
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:33867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:41795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:47.904789 14593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:47.905117 14593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:47.905192 14593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:47.910981 14593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:47.911124 14593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:47.911167 14593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:47.911216 14593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:47.911259 14593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:47.918915 14593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:33867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:41795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:41795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:47.920261 14593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:47.921685 14593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:47.931684 14599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:47.931807 14601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:47.931691 14608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:47.931947 14593 server_base.cc:1034] running on GCE node
I20260808 04:39:47.933068 14593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:47.934309 14593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:47.935629 14593 hybrid_clock.cc:648] HybridClock initialized: now 1786163987935570 us; error 92 us; skew 500 ppm
Aug 08 04:39:47 dist-test-slave-drl0 krb5kdc[14573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163987, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:47.939656 14593 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:39:47.941192 14593 webserver.cc:460] Webserver started at http://127.15.228.126:38077/ using document root <none> and password file <none>
I20260808 04:39:47.941932 14593 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:47.942188 14593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:47.942512 14593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:47.944959 14593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "09d7d16af3084b9d9a5db5050cff1254"
format_stamp: "Formatted at 2026-08-08 04:39:47 on dist-test-slave-drl0"
server_key: "32839eb95a9bd320055861067970abee"
server_key_iv: "9968dce73428e0b609d9daadbf4fdc34"
server_key_version: "encryptionkey@0"
I20260808 04:39:47.945601 14593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "09d7d16af3084b9d9a5db5050cff1254"
format_stamp: "Formatted at 2026-08-08 04:39:47 on dist-test-slave-drl0"
server_key: "32839eb95a9bd320055861067970abee"
server_key_iv: "9968dce73428e0b609d9daadbf4fdc34"
server_key_version: "encryptionkey@0"
I20260808 04:39:47.950881 14593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260808 04:39:47.953780 14617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:47.955597 14593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 04:39:47.955789 14593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "09d7d16af3084b9d9a5db5050cff1254"
format_stamp: "Formatted at 2026-08-08 04:39:47 on dist-test-slave-drl0"
server_key: "32839eb95a9bd320055861067970abee"
server_key_iv: "9968dce73428e0b609d9daadbf4fdc34"
server_key_version: "encryptionkey@0"
I20260808 04:39:47.955933 14593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:47.968086 14593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:47.971546 14593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:47.971812 14593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:47.981500 14593 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:41795
I20260808 04:39:47.981514 14688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:41795 every 8 connection(s)
I20260808 04:39:47.983255 14593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:47.986305 14689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:47.992429 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 14593
I20260808 04:39:47.992597 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:47.992690 14689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254: Bootstrap starting.
I20260808 04:39:47.992940 16273 external_mini_cluster.cc:1428] Setting key 18a9b49370b1f90a2f724b2c535a81c4
I20260808 04:39:47.996023 14689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:47.996850 14689 log.cc:824] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:47.999389 14689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254: No bootstrap required, opened a new log
I20260808 04:39:48.004217 14689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d7d16af3084b9d9a5db5050cff1254" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41795 } }
I20260808 04:39:48.004462 14689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:48.004506 14689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09d7d16af3084b9d9a5db5050cff1254, State: Initialized, Role: FOLLOWER
Aug 08 04:39:48 dist-test-slave-drl0 krb5kdc[14573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:39:48.005172 14689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d7d16af3084b9d9a5db5050cff1254" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41795 } }
I20260808 04:39:48.005363 14689 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:48.005450 14689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:48.005559 14689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:48.007927 14689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d7d16af3084b9d9a5db5050cff1254" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41795 } }
I20260808 04:39:48.008519 14689 leader_election.cc:302] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09d7d16af3084b9d9a5db5050cff1254; no voters:
I20260808 04:39:48.009083 14689 leader_election.cc:288] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:48.009286 14695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:48.009609 14695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254 [term 1 LEADER]: Becoming Leader. State: Replica: 09d7d16af3084b9d9a5db5050cff1254, State: Running, Role: LEADER
I20260808 04:39:48.010046 14695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d7d16af3084b9d9a5db5050cff1254" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41795 } }
I20260808 04:39:48.011099 14689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:48.012017 14696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09d7d16af3084b9d9a5db5050cff1254" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d7d16af3084b9d9a5db5050cff1254" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41795 } } }
I20260808 04:39:48.012156 14696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:48.012980 14697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09d7d16af3084b9d9a5db5050cff1254. Latest consensus state: current_term: 1 leader_uuid: "09d7d16af3084b9d9a5db5050cff1254" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d7d16af3084b9d9a5db5050cff1254" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 41795 } } }
I20260808 04:39:48.013093 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:47.994414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34555 (local address 127.15.228.126:41795)
0808 04:39:47.994990 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:47.995008 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:47.995044 (+ 36us) server_negotiation.cc:407] Connection header received
0808 04:39:47.995883 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:47.995922 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:47.996721 (+ 799us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:47.997248 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:47.998139 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:47.999709 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:48.001020 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:48.001667 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:48.001828 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:48.005491 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:48.005511 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:48.005532 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:48.005571 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:48.008749 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:48.009477 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:48.009486 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:48.009494 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:48.009653 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:48.010029 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:48.010033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:48.010035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:48.010761 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:48.011045 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:48.011527 (+ 482us) server_negotiation.cc:299] Negotiation successful
0808 04:39:48.011924 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":247,"threads_started":1}
I20260808 04:39:48.013139 14701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:48.013137 14697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:48.016860 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:39:48.017309 14701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:48.023993 14701 catalog_manager.cc:1347] Generated new cluster ID: fb11c4f25ef045f9a42c1c3fbf08b20c
I20260808 04:39:48.024070 14701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:48.033141 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:48.022823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47617 (local address 127.15.228.126:41795)
0808 04:39:48.023004 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:48.023009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:48.023676 (+ 667us) server_negotiation.cc:407] Connection header received
0808 04:39:48.023924 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:48.023927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:48.023984 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:48.024041 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:48.025058 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:48.025919 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:48.027090 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:48.027491 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:48.027603 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:48.029375 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:48.029379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:48.029382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:48.029406 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:48.031066 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:48.031507 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:48.031511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:48.031513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:48.031587 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:48.031843 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:48.031845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:48.031847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:48.032082 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:48.032205 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:48.032827 (+ 622us) server_negotiation.cc:299] Negotiation successful
0808 04:39:48.032934 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 04:39:48.052985 14701 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:48.054095 14701 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:48.062410 14701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09d7d16af3084b9d9a5db5050cff1254: Generated new TSK 0
I20260808 04:39:48.063405 14701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:48.072713 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 14593
2026-08-08T04:39:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1771 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:48 dist-test-slave-drl0 krb5kdc[14730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:48 dist-test-slave-drl0 krb5kdc[14730](info): set up 2 sockets
Aug 08 04:39:48 dist-test-slave-drl0 krb5kdc[14730](info): commencing operation
krb5kdc: starting...
W20260808 04:39:49.373184 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.269s user 0.003s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:49 dist-test-slave-drl0 krb5kdc[14730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:39:49.520408 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:43635
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:42695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:43635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:49.647289 14752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:49.647575 14752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:49.647621 14752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:49.652099 14752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:49.652165 14752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:49.652186 14752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:49.652204 14752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:49.652223 14752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:49.657687 14752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:42695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:43635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:43635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:49.658807 14752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:49.659914 14752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:49.669438 14761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:49.669539 14752 server_base.cc:1034] running on GCE node
W20260808 04:39:49.669445 14759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:49.669458 14763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:49.670440 14752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:49.671584 14752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:49.672806 14752 hybrid_clock.cc:648] HybridClock initialized: now 1786163989672780 us; error 40 us; skew 500 ppm
Aug 08 04:39:49 dist-test-slave-drl0 krb5kdc[14730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163989, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:49.681748 14752 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:39:49.683213 14752 webserver.cc:460] Webserver started at http://127.15.228.126:41267/ using document root <none> and password file <none>
I20260808 04:39:49.683979 14752 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:49.684099 14752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:49.684398 14752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:49.687294 14752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "2855ccee04bc4bf8badc19dda28dc37a"
format_stamp: "Formatted at 2026-08-08 04:39:49 on dist-test-slave-drl0"
server_key: "e56fd6dcb4243b179ac4ce45c09ff08b"
server_key_iv: "a5902ee170e0b70c4382e77098d54ae9"
server_key_version: "encryptionkey@0"
I20260808 04:39:49.688090 14752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "2855ccee04bc4bf8badc19dda28dc37a"
format_stamp: "Formatted at 2026-08-08 04:39:49 on dist-test-slave-drl0"
server_key: "e56fd6dcb4243b179ac4ce45c09ff08b"
server_key_iv: "a5902ee170e0b70c4382e77098d54ae9"
server_key_version: "encryptionkey@0"
I20260808 04:39:49.693420 14752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 04:39:49.696486 14774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:49.698138 14752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 04:39:49.698336 14752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "2855ccee04bc4bf8badc19dda28dc37a"
format_stamp: "Formatted at 2026-08-08 04:39:49 on dist-test-slave-drl0"
server_key: "e56fd6dcb4243b179ac4ce45c09ff08b"
server_key_iv: "a5902ee170e0b70c4382e77098d54ae9"
server_key_version: "encryptionkey@0"
I20260808 04:39:49.698487 14752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:49.714308 14752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:49.719122 14752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:49.719388 14752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:49.728180 14752 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:43635
I20260808 04:39:49.728259 14844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:43635 every 8 connection(s)
I20260808 04:39:49.729212 14752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:49.729979 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 14752
I20260808 04:39:49.730149 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:49.730446 16273 external_mini_cluster.cc:1428] Setting key cf45fcf69e0e113db0eee46feab5daa1
I20260808 04:39:49.733881 14846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:49.741390 14846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a: Bootstrap starting.
I20260808 04:39:49.741443 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:49.731883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51599 (local address 127.15.228.126:43635)
0808 04:39:49.732317 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:49.732334 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:49.732371 (+ 37us) server_negotiation.cc:407] Connection header received
0808 04:39:49.733025 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:49.733041 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:49.733381 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:49.733662 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:49.734230 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:49.735536 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:49.736902 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:49.737412 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:49.737544 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:49.738561 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:49.738566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:49.738578 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:49.739190 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:49.739307 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:49.739353 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:49.739504 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:49.739790 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 04:39:49.740193 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":135,"threads_started":1}
I20260808 04:39:49.743978 14846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:49.745010 14846 log.cc:824] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:49.746151 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:39:49.747511 14846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a: No bootstrap required, opened a new log
I20260808 04:39:49.751156 14846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2855ccee04bc4bf8badc19dda28dc37a" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43635 } }
I20260808 04:39:49.751410 14846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:49.751499 14846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2855ccee04bc4bf8badc19dda28dc37a, State: Initialized, Role: FOLLOWER
I20260808 04:39:49.752046 14846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2855ccee04bc4bf8badc19dda28dc37a" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43635 } }
I20260808 04:39:49.752238 14846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:49.752321 14846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:49.752454 14846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:49.753863 14846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2855ccee04bc4bf8badc19dda28dc37a" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43635 } }
I20260808 04:39:49.754104 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:49.748538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60007 (local address 127.15.228.126:43635)
0808 04:39:49.748677 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:49.748682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:49.748798 (+ 116us) server_negotiation.cc:407] Connection header received
0808 04:39:49.748973 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:49.748976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:49.749045 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:49.749143 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:49.749445 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:49.750584 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:49.751629 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:49.752008 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:49.752137 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:49.753234 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:49.753238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:49.753241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:49.753255 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:49.753327 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:49.753344 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:49.753434 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:49.753770 (+ 336us) server_negotiation.cc:299] Negotiation successful
0808 04:39:49.753845 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 04:39:49.754663 14846 leader_election.cc:302] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2855ccee04bc4bf8badc19dda28dc37a; no voters:
I20260808 04:39:49.755343 14846 leader_election.cc:288] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:49.755502 14860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:49.755956 14860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a [term 1 LEADER]: Becoming Leader. State: Replica: 2855ccee04bc4bf8badc19dda28dc37a, State: Running, Role: LEADER
I20260808 04:39:49.756279 14860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2855ccee04bc4bf8badc19dda28dc37a" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43635 } }
I20260808 04:39:49.756728 14846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:49.758152 14861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2855ccee04bc4bf8badc19dda28dc37a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2855ccee04bc4bf8badc19dda28dc37a" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43635 } } }
I20260808 04:39:49.758284 14861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:49.758728 14872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:49.758674 14863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2855ccee04bc4bf8badc19dda28dc37a. Latest consensus state: current_term: 1 leader_uuid: "2855ccee04bc4bf8badc19dda28dc37a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2855ccee04bc4bf8badc19dda28dc37a" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 43635 } } }
I20260808 04:39:49.758836 14863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:49.762107 14872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:49.768198 14872 catalog_manager.cc:1347] Generated new cluster ID: 51af04404b18491f9663772af851367c
I20260808 04:39:49.768265 14872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:49.776837 14872 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:49.778138 14872 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:49.798107 14872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2855ccee04bc4bf8badc19dda28dc37a: Generated new TSK 0
I20260808 04:39:49.799150 14872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:49.810982 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 14752
2026-08-08T04:39:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1736 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:49 dist-test-slave-drl0 krb5kdc[14884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:49 dist-test-slave-drl0 krb5kdc[14884](info): set up 2 sockets
Aug 08 04:39:49 dist-test-slave-drl0 krb5kdc[14884](info): commencing operation
krb5kdc: starting...
W20260808 04:39:51.107246 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.266s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:51 dist-test-slave-drl0 krb5kdc[14884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:39:51.255493 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:40029
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:38555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:40029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:51.383185 14904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:51.383466 14904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:51.383508 14904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:51.387627 14904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:51.387702 14904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:51.387723 14904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:51.387742 14904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:51.387758 14904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:51.393072 14904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:38555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:40029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:40029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:51.394179 14904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:51.395248 14904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:51.404510 14909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:51.404510 14912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:51.404510 14910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:51.404922 14904 server_base.cc:1034] running on GCE node
I20260808 04:39:51.405524 14904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:51.406698 14904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:51.407920 14904 hybrid_clock.cc:648] HybridClock initialized: now 1786163991407902 us; error 40 us; skew 500 ppm
Aug 08 04:39:51 dist-test-slave-drl0 krb5kdc[14884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163991, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:51.411551 14904 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:39:51.413079 14904 webserver.cc:460] Webserver started at http://127.15.228.126:39667/ using document root <none> and password file <none>
I20260808 04:39:51.413923 14904 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:51.414052 14904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:51.414547 14904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:51.417870 14904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "1f8c74a4c56b44a8aa15a50a6181c4f1"
format_stamp: "Formatted at 2026-08-08 04:39:51 on dist-test-slave-drl0"
server_key: "5babc41916fa77934541690154bef8b1"
server_key_iv: "d63794c40aed777004f8cd8a23af550f"
server_key_version: "encryptionkey@0"
I20260808 04:39:51.418764 14904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "1f8c74a4c56b44a8aa15a50a6181c4f1"
format_stamp: "Formatted at 2026-08-08 04:39:51 on dist-test-slave-drl0"
server_key: "5babc41916fa77934541690154bef8b1"
server_key_iv: "d63794c40aed777004f8cd8a23af550f"
server_key_version: "encryptionkey@0"
I20260808 04:39:51.424520 14904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 04:39:51.427852 14919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:51.429324 14904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 04:39:51.429502 14904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "1f8c74a4c56b44a8aa15a50a6181c4f1"
format_stamp: "Formatted at 2026-08-08 04:39:51 on dist-test-slave-drl0"
server_key: "5babc41916fa77934541690154bef8b1"
server_key_iv: "d63794c40aed777004f8cd8a23af550f"
server_key_version: "encryptionkey@0"
I20260808 04:39:51.429643 14904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:51.435089 14904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:51.438488 14904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:51.438709 14904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:51.446813 14904 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:40029
I20260808 04:39:51.446858 14997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:40029 every 8 connection(s)
I20260808 04:39:51.447842 14904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:51.450896 14998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:51.454144 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 14904
I20260808 04:39:51.454351 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:51.454694 16273 external_mini_cluster.cc:1428] Setting key 7181ee333cd05db96f6b432b7e94d29b
I20260808 04:39:51.458575 14998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1: Bootstrap starting.
I20260808 04:39:51.461380 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:51.457098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54783 (local address 127.15.228.126:40029)
0808 04:39:51.457639 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:51.457654 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:51.457694 (+ 40us) server_negotiation.cc:407] Connection header received
0808 04:39:51.458544 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:51.458563 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:51.458912 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:51.459330 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:51.460153 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:51.460157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:51.460169 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:51.460180 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:51.460234 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:51.460254 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:51.460418 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:51.460489 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 04:39:51.460646 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":144,"threads_started":1}
I20260808 04:39:51.461416 14998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:51.462270 14998 log.cc:824] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:51.464833 14998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1: No bootstrap required, opened a new log
I20260808 04:39:51.464941 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:39:51.469538 14998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8c74a4c56b44a8aa15a50a6181c4f1" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 40029 } }
I20260808 04:39:51.469810 14998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 04:39:51.469830 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:51.467198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37757 (local address 127.15.228.126:40029)
0808 04:39:51.467497 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:51.467506 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:51.467939 (+ 433us) server_negotiation.cc:407] Connection header received
0808 04:39:51.468033 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:51.468035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:51.468068 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:51.468139 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:51.469141 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:51.469145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:51.469148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 04:39:51.469166 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:51.469217 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 04:39:51.469234 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:51.469356 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:51.469534 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 04:39:51.469628 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260808 04:39:51.469873 14998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f8c74a4c56b44a8aa15a50a6181c4f1, State: Initialized, Role: FOLLOWER
I20260808 04:39:51.470325 14998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8c74a4c56b44a8aa15a50a6181c4f1" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 40029 } }
I20260808 04:39:51.470494 14998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 04:39:51.470553 14998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 04:39:51.470642 14998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:51.471760 14998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8c74a4c56b44a8aa15a50a6181c4f1" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 40029 } }
I20260808 04:39:51.472065 14998 leader_election.cc:302] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f8c74a4c56b44a8aa15a50a6181c4f1; no voters:
I20260808 04:39:51.472349 14998 leader_election.cc:288] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 04:39:51.472471 15010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:51.472721 15010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 1f8c74a4c56b44a8aa15a50a6181c4f1, State: Running, Role: LEADER
I20260808 04:39:51.473071 15010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8c74a4c56b44a8aa15a50a6181c4f1" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 40029 } }
I20260808 04:39:51.473630 14998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:51.474916 15014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f8c74a4c56b44a8aa15a50a6181c4f1. Latest consensus state: current_term: 1 leader_uuid: "1f8c74a4c56b44a8aa15a50a6181c4f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8c74a4c56b44a8aa15a50a6181c4f1" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 40029 } } }
I20260808 04:39:51.474984 15011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f8c74a4c56b44a8aa15a50a6181c4f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8c74a4c56b44a8aa15a50a6181c4f1" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 40029 } } }
I20260808 04:39:51.475077 15014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:51.475080 15011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:51.475454 15024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:51.480726 15024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:51.488368 15024 catalog_manager.cc:1347] Generated new cluster ID: 01f69ca6994b458884c4fc2870658645
I20260808 04:39:51.488504 15024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:51.500556 15024 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:51.501184 15024 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:51.506753 15024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f8c74a4c56b44a8aa15a50a6181c4f1: Generated new TSK 0
I20260808 04:39:51.507609 15024 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 04:39:51.516151 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 14904
2026-08-08T04:39:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1704 ms)
[----------] 18 tests from AuthTokenIssuingTest (31357 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:51 dist-test-slave-drl0 krb5kdc[15043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:51 dist-test-slave-drl0 krb5kdc[15043](info): set up 2 sockets
Aug 08 04:39:51 dist-test-slave-drl0 krb5kdc[15043](info): commencing operation
krb5kdc: starting...
W20260808 04:39:52.906141 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.361s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:53 dist-test-slave-drl0 krb5kdc[15043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:39:53.136984 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:42957
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:42957,127.15.228.125:38973,127.15.228.124:40631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:53.359246 15060 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:39:53.359825 15060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:53.359879 15060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:53.360113 15060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:53.367784 15060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:53.367909 15060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:53.367954 15060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:53.367980 15060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:53.368062 15060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:53.377725 15060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:42957,127.15.228.125:38973,127.15.228.124:40631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:42957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:53.379983 15060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:53.382005 15060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:53.397675 15066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:53.397770 15069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:53.397670 15067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:53.398315 15060 server_base.cc:1034] running on GCE node
I20260808 04:39:53.399391 15060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:53.401618 15060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:53.402906 15060 hybrid_clock.cc:648] HybridClock initialized: now 1786163993402867 us; error 71 us; skew 500 ppm
Aug 08 04:39:53 dist-test-slave-drl0 krb5kdc[15043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163993, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:53.408264 15060 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:39:53.409933 15060 webserver.cc:460] Webserver started at http://127.15.228.126:39335/ using document root <none> and password file <none>
I20260808 04:39:53.411068 15060 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:53.411258 15060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:53.411662 15060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:53.415710 15060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "397576f71e4b4429a6fb93d48e30643b"
format_stamp: "Formatted at 2026-08-08 04:39:53 on dist-test-slave-drl0"
server_key: "ac92e91a392aba9b09969452ec8f93f0"
server_key_iv: "6cd074c81564a2fb947e2600364d027c"
server_key_version: "encryptionkey@0"
I20260808 04:39:53.416883 15060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "397576f71e4b4429a6fb93d48e30643b"
format_stamp: "Formatted at 2026-08-08 04:39:53 on dist-test-slave-drl0"
server_key: "ac92e91a392aba9b09969452ec8f93f0"
server_key_iv: "6cd074c81564a2fb947e2600364d027c"
server_key_version: "encryptionkey@0"
I20260808 04:39:53.424641 15060 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260808 04:39:53.429733 15082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:53.432984 15060 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260808 04:39:53.433305 15060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "397576f71e4b4429a6fb93d48e30643b"
format_stamp: "Formatted at 2026-08-08 04:39:53 on dist-test-slave-drl0"
server_key: "ac92e91a392aba9b09969452ec8f93f0"
server_key_iv: "6cd074c81564a2fb947e2600364d027c"
server_key_version: "encryptionkey@0"
I20260808 04:39:53.433557 15060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:53.471551 15060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:53.477772 15060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:53.478132 15060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:53.494750 15060 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:42957
I20260808 04:39:53.495427 15152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:42957 every 8 connection(s)
I20260808 04:39:53.496656 15060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:53.501809 15153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:53.504405 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 15060
I20260808 04:39:53.504670 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:53.505304 16273 external_mini_cluster.cc:1428] Setting key 86b8c330130090b123bcbe78c6a5b9da
I20260808 04:39:53.509815 15153 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.228.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.125@KRBTEST.COM" created.
Aug 08 04:39:53 dist-test-slave-drl0 krb5kdc[15043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163993, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:39:53.545892 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:53.515061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:42957 (local address 127.0.0.1:35986)
0808 04:39:53.515895 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:53.515999 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:53.517489 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:53.519140 (+ 1651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:53.519149 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:53.519805 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:53.520809 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:53.520871 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:53.524077 (+ 3206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:53.524092 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:53.525576 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:53.525588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:53.527779 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:53.527785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:53.528046 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:53.529069 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:53.529116 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:53.533315 (+ 4199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:53.538456 (+ 5141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:53.538474 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:53.538492 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:53.538875 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:53.540499 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:53.540502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:53.540503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:53.540856 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:53.542481 (+ 1625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:53.542496 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:53.542683 (+ 187us) client_negotiation.cc:769] Sending connection context
0808 04:39:53.543013 (+ 330us) client_negotiation.cc:240] Negotiation successful
0808 04:39:53.543621 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":76,"thread_start_us":242,"threads_started":1}
I20260808 04:39:53.545892 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:53.515062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35986 (local address 127.15.228.126:42957)
0808 04:39:53.516041 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:53.516071 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:53.516178 (+ 107us) server_negotiation.cc:407] Connection header received
0808 04:39:53.518058 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:53.518079 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:53.518580 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:53.519047 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:53.521318 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:53.523763 (+ 2445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:53.526004 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:53.527436 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:53.527679 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:53.533565 (+ 5886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:53.533588 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:53.533646 (+ 58us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:53.533714 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:53.538207 (+ 4493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:53.539619 (+ 1412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:53.539635 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:53.539650 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:53.540276 (+ 626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:53.541079 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:53.541095 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:53.541099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:53.542215 (+ 1116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:53.542504 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:53.543143 (+ 639us) server_negotiation.cc:299] Negotiation successful
0808 04:39:53.543698 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":264,"server-negotiator.queue_time_us":740,"thread_start_us":255,"threads_started":1}
I20260808 04:39:53.550894 15153 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:39:53.552508 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:53.551835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38973 (local address 127.0.0.1:59320)
0808 04:39:53.552025 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:53.552163 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260808 04:39:53.553035 15086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38973: connect: Connection refused (error 111)
I20260808 04:39:53.555109 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:53.554684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38973 (local address 127.0.0.1:59330)
0808 04:39:53.554850 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:53.554918 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260808 04:39:53.555334 15153 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.125:38973: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38973: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.125.keytab.
Entry for principal kudu/127.15.228.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.125.keytab.
I20260808 04:39:53.560570 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.125:38973
--webserver_interface=127.15.228.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.125.keytab
--principal=kudu/127.15.228.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:42957,127.15.228.125:38973,127.15.228.124:40631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:39:53.604830 15153 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } attempt: 1
I20260808 04:39:53.606395 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:53.605950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38973 (local address 127.0.0.1:59344)
0808 04:39:53.606206 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:53.606287 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260808 04:39:53.608091 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:53.607577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38973 (local address 127.0.0.1:59354)
0808 04:39:53.607813 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:53.607929 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260808 04:39:53.608377 15153 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.125:38973: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38973: connect: Connection refused (error 111)
I20260808 04:39:53.676692 15153 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } attempt: 2
I20260808 04:39:53.678715 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:53.678054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38973 (local address 127.0.0.1:59356)
0808 04:39:53.678349 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:53.678455 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260808 04:39:53.679984 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:53.679608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38973 (local address 127.0.0.1:59368)
0808 04:39:53.679759 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:53.679866 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260808 04:39:53.680171 15153 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.125:38973: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38973: connect: Connection refused (error 111)
I20260808 04:39:53.771378 15153 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } attempt: 3
I20260808 04:39:53.772848 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:53.772293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38973 (local address 127.0.0.1:59380)
0808 04:39:53.772613 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:53.772728 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260808 04:39:53.773911 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:53.773562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38973 (local address 127.0.0.1:59386)
0808 04:39:53.773750 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:53.773829 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260808 04:39:53.774093 15153 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.125:38973: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38973: connect: Connection refused (error 111)
W20260808 04:39:53.782292 15164 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:39:53.782975 15164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:53.783121 15164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:53.783319 15164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:53.791118 15164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:53.791298 15164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:53.791392 15164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:53.791494 15164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:53.791570 15164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:53.803509 15164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:42957,127.15.228.125:38973,127.15.228.124:40631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.125.keytab
--principal=kudu/127.15.228.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.125:38973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.228.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:53.806082 15164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:53.808554 15164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:53.824815 15173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:53.824833 15171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:53.824877 15170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:53.826339 15164 server_base.cc:1034] running on GCE node
I20260808 04:39:53.827831 15164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:53.830278 15164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:53.833801 15164 hybrid_clock.cc:648] HybridClock initialized: now 1786163993833726 us; error 74 us; skew 500 ppm
Aug 08 04:39:53 dist-test-slave-drl0 krb5kdc[15043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163993, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:53.840339 15164 init.cc:377] Logged in from keytab as kudu/127.15.228.125@KRBTEST.COM (short username kudu)
I20260808 04:39:53.842530 15164 webserver.cc:460] Webserver started at http://127.15.228.125:39853/ using document root <none> and password file <none>
I20260808 04:39:53.843930 15164 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:53.844099 15164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:53.844544 15164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:53.849535 15164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-1/data/instance:
uuid: "c7fec59dcdf44b63ab3858550fa6e745"
format_stamp: "Formatted at 2026-08-08 04:39:53 on dist-test-slave-drl0"
server_key: "ab8e86164b6d4474d1126bb9ea2e01d6"
server_key_iv: "1b8be5c9a446d2e0d93f52e2c3172a67"
server_key_version: "encryptionkey@0"
I20260808 04:39:53.850656 15164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-1/wal/instance:
uuid: "c7fec59dcdf44b63ab3858550fa6e745"
format_stamp: "Formatted at 2026-08-08 04:39:53 on dist-test-slave-drl0"
server_key: "ab8e86164b6d4474d1126bb9ea2e01d6"
server_key_iv: "1b8be5c9a446d2e0d93f52e2c3172a67"
server_key_version: "encryptionkey@0"
I20260808 04:39:53.861137 15164 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.001s
I20260808 04:39:53.867009 15185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:53.869683 15164 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260808 04:39:53.869910 15164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-1/wal
uuid: "c7fec59dcdf44b63ab3858550fa6e745"
format_stamp: "Formatted at 2026-08-08 04:39:53 on dist-test-slave-drl0"
server_key: "ab8e86164b6d4474d1126bb9ea2e01d6"
server_key_iv: "1b8be5c9a446d2e0d93f52e2c3172a67"
server_key_version: "encryptionkey@0"
I20260808 04:39:53.870110 15164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:53.903916 15164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:53.910245 15164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:53.910560 15164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:53.917352 15153 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } attempt: 4
I20260808 04:39:53.918888 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:53.918396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38973 (local address 127.0.0.1:59398)
0808 04:39:53.918671 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:53.918781 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260808 04:39:53.920854 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:53.920328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38973 (local address 127.0.0.1:59408)
0808 04:39:53.920564 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:53.920772 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260808 04:39:53.921079 15153 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.125:38973: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38973: connect: Connection refused (error 111)
I20260808 04:39:53.926494 15164 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.125:38973
I20260808 04:39:53.927152 15257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.125:38973 every 8 connection(s)
I20260808 04:39:53.929317 15164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-1/data/info.pb
I20260808 04:39:53.934582 15258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:53.936473 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 15164
I20260808 04:39:53.936654 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-1/wal/instance
I20260808 04:39:53.937052 16273 external_mini_cluster.cc:1428] Setting key 81a4ac3c61476e5efb384193c0042bfc
I20260808 04:39:53.940148 15258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.228.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.124@KRBTEST.COM" created.
Aug 08 04:39:53 dist-test-slave-drl0 krb5kdc[15043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163993, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.125@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:39:53.970721 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:53.943621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.15.228.126:42957)
0808 04:39:53.943935 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:53.943941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:53.945309 (+ 1368us) server_negotiation.cc:407] Connection header received
0808 04:39:53.947016 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:53.947023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:53.947124 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:53.947289 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:53.950071 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:53.951521 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:53.954243 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:53.954797 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:53.955118 (+ 321us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:53.960775 (+ 5657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:53.960787 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:53.960789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:53.960833 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:53.965080 (+ 4247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:53.966571 (+ 1491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:53.966579 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:53.966583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:53.966777 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:53.967769 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:53.967776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:53.967792 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:53.968298 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:53.968520 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:53.969934 (+ 1414us) server_negotiation.cc:299] Negotiation successful
0808 04:39:53.970276 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260808 04:39:53.972354 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:53.944195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:42957 (local address 127.0.0.1:35994)
0808 04:39:53.944983 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:53.945095 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:53.946537 (+ 1442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:53.947520 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:53.947541 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:53.948257 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:53.949544 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:53.949611 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:53.951914 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:53.951926 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:53.953780 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:53.953802 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:53.955243 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:53.955251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:53.955919 (+ 668us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:53.957308 (+ 1389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:53.957329 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:53.960440 (+ 3111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:53.965428 (+ 4988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:53.965434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:53.965461 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:53.966304 (+ 843us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:53.967242 (+ 938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:53.967246 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:53.967248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:53.967586 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:53.968673 (+ 1087us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:53.968692 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:53.969445 (+ 753us) client_negotiation.cc:769] Sending connection context
0808 04:39:53.969847 (+ 402us) client_negotiation.cc:240] Negotiation successful
0808 04:39:53.970429 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":248,"threads_started":1}
I20260808 04:39:53.975450 15258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:39:53 dist-test-slave-drl0 krb5kdc[15043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163993, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.125@KRBTEST.COM for kudu/127.15.228.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.124.keytab.
Entry for principal kudu/127.15.228.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.124.keytab.
I20260808 04:39:53.992763 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.124:40631
--webserver_interface=127.15.228.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.124.keytab
--principal=kudu/127.15.228.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:42957,127.15.228.125:38973,127.15.228.124:40631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:39:53.993887 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:53.976506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.15.228.125:38973)
0808 04:39:53.976966 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:53.976977 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:53.977004 (+ 27us) server_negotiation.cc:407] Connection header received
0808 04:39:53.977072 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:53.977077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:53.977300 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:53.977605 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:53.978189 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:53.979842 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:53.981312 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:53.981974 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:53.982232 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:53.985494 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:53.985520 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:53.985564 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:53.985609 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:53.988799 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:53.989660 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:53.989668 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:53.989669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:53.989831 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:53.990403 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:53.990407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:53.990408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:53.991136 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:53.991389 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:53.993432 (+ 2043us) server_negotiation.cc:299] Negotiation successful
0808 04:39:53.993642 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":218,"threads_started":1}
I20260808 04:39:53.993856 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:53.976295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38973 (local address 127.0.0.1:59416)
0808 04:39:53.976512 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:53.976546 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:53.976656 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:53.977523 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:53.977527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:53.977852 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:53.977968 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:53.977984 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:53.980261 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:53.980265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:53.981110 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:53.981128 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:53.982216 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:53.982231 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:53.982413 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:53.983320 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:53.983333 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:53.985210 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:53.989008 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:53.989025 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:53.989027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:53.989392 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:53.989997 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:53.990000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:53.990001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:53.990130 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:53.992777 (+ 2647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:53.992781 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:53.992968 (+ 187us) client_negotiation.cc:769] Sending connection context
0808 04:39:53.993265 (+ 297us) client_negotiation.cc:240] Negotiation successful
0808 04:39:53.993542 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260808 04:39:53.998306 15258 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:39:53.999553 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:53.999136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:40631 (local address 127.0.0.1:46668)
0808 04:39:53.999331 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:53.999466 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260808 04:39:53.999989 15186 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.228.124:40631: connect: Connection refused (error 111)
I20260808 04:39:54.002274 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.001471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:40631 (local address 127.0.0.1:46682)
0808 04:39:54.001783 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:54.001936 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260808 04:39:54.002671 15258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.124:40631: Network error: Client connection negotiation failed: client connection to 127.15.228.124:40631: connect: Connection refused (error 111)
I20260808 04:39:54.052265 15258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } attempt: 1
I20260808 04:39:54.054033 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.053439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:40631 (local address 127.0.0.1:46688)
0808 04:39:54.053782 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:54.053894 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260808 04:39:54.055796 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.055016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:40631 (local address 127.0.0.1:46690)
0808 04:39:54.055322 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:54.055569 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260808 04:39:54.056123 15258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.124:40631: Network error: Client connection negotiation failed: client connection to 127.15.228.124:40631: connect: Connection refused (error 111)
I20260808 04:39:54.124390 15258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } attempt: 2
I20260808 04:39:54.126187 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.125548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:40631 (local address 127.0.0.1:46698)
0808 04:39:54.125845 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:54.126005 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260808 04:39:54.127627 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.127228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:40631 (local address 127.0.0.1:46702)
0808 04:39:54.127420 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:54.127519 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260808 04:39:54.127883 15258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.124:40631: Network error: Client connection negotiation failed: client connection to 127.15.228.124:40631: connect: Connection refused (error 111)
W20260808 04:39:54.219101 15264 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260808 04:39:54.219440 15258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } attempt: 3
W20260808 04:39:54.219617 15264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:54.219713 15264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:54.219889 15264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 04:39:54.220307 15153 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } attempt: 5
I20260808 04:39:54.221122 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.220463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:40631 (local address 127.0.0.1:46712)
0808 04:39:54.220791 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:54.220911 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260808 04:39:54.222553 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.222068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:40631 (local address 127.0.0.1:46714)
0808 04:39:54.222368 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:54.222446 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
W20260808 04:39:54.222759 15258 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.124:40631: Network error: Client connection negotiation failed: client connection to 127.15.228.124:40631: connect: Connection refused (error 111)
W20260808 04:39:54.227285 15264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:54.227432 15264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:54.227545 15264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:54.227628 15264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:54.227694 15264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
Aug 08 04:39:54 dist-test-slave-drl0 krb5kdc[15043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163993, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for kudu/127.15.228.125@KRBTEST.COM
I20260808 04:39:54.237459 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.221229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59422 (local address 127.15.228.125:38973)
0808 04:39:54.221455 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.221460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.221505 (+ 45us) server_negotiation.cc:407] Connection header received
0808 04:39:54.221759 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.221763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.221867 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.222084 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.222910 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.224519 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.225995 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.226463 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.226836 (+ 373us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:54.230780 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:54.230791 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:54.230794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:54.230843 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:54.233867 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.235076 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.235079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.235081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.235222 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.235815 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.235827 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.235828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.236176 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:54.236359 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:54.237019 (+ 660us) server_negotiation.cc:299] Negotiation successful
0808 04:39:54.237218 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 04:39:54.237484 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.221108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38973 (local address 127.0.0.1:59422)
0808 04:39:54.221399 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:54.221425 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:54.221594 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:54.222099 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:54.222113 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:54.222443 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:54.222612 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:54.222652 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.224749 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.224756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:54.225707 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:54.225723 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.226738 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.226839 (+ 101us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:54.227107 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:54.228172 (+ 1065us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:54.228200 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:54.230536 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:54.234164 (+ 3628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:54.234187 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:54.234190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:54.234753 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:54.235425 (+ 672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:54.235429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:54.235431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:54.235624 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:54.236388 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:54.236391 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:54.236556 (+ 165us) client_negotiation.cc:769] Sending connection context
0808 04:39:54.236880 (+ 324us) client_negotiation.cc:240] Negotiation successful
0808 04:39:54.237182 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260808 04:39:54.239094 15153 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:39:54.240379 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.239831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:40631 (local address 127.0.0.1:46726)
0808 04:39:54.240066 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:54.240201 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 04:39:54.241117 15264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:42957,127.15.228.125:38973,127.15.228.124:40631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.124.keytab
--principal=kudu/127.15.228.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.124:40631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.228.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:54.241804 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.241329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:40631 (local address 127.0.0.1:46740)
0808 04:39:54.241589 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:54.241664 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260808 04:39:54.242004 15153 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.124:40631: Network error: Client connection negotiation failed: client connection to 127.15.228.124:40631: connect: Connection refused (error 111)
I20260808 04:39:54.243721 15264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:54.246698 15264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:54.260982 15271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:54.261222 15264 server_base.cc:1034] running on GCE node
W20260808 04:39:54.261021 15274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:54.261612 15270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:54.262645 15264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:54.264698 15264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:54.268137 15264 hybrid_clock.cc:648] HybridClock initialized: now 1786163994268104 us; error 69 us; skew 500 ppm
Aug 08 04:39:54 dist-test-slave-drl0 krb5kdc[15043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163994, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:54.273020 15264 init.cc:377] Logged in from keytab as kudu/127.15.228.124@KRBTEST.COM (short username kudu)
I20260808 04:39:54.274541 15264 webserver.cc:460] Webserver started at http://127.15.228.124:34985/ using document root <none> and password file <none>
I20260808 04:39:54.275532 15264 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:54.275647 15264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:54.276021 15264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:54.279877 15264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-2/data/instance:
uuid: "9c718b7c8b2048c085c92fe65d36354e"
format_stamp: "Formatted at 2026-08-08 04:39:54 on dist-test-slave-drl0"
server_key: "7a23f85398f4628d8312bf92c3daef8f"
server_key_iv: "7fa096998c9a91b71d14c00e9d1354d1"
server_key_version: "encryptionkey@0"
I20260808 04:39:54.280969 15264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-2/wal/instance:
uuid: "9c718b7c8b2048c085c92fe65d36354e"
format_stamp: "Formatted at 2026-08-08 04:39:54 on dist-test-slave-drl0"
server_key: "7a23f85398f4628d8312bf92c3daef8f"
server_key_iv: "7fa096998c9a91b71d14c00e9d1354d1"
server_key_version: "encryptionkey@0"
I20260808 04:39:54.288791 15264 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260808 04:39:54.293251 15153 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } attempt: 1
I20260808 04:39:54.295234 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.294427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:40631 (local address 127.0.0.1:46756)
0808 04:39:54.294858 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:54.295094 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
I20260808 04:39:54.295953 15283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:54.296967 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.296522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:40631 (local address 127.0.0.1:46772)
0808 04:39:54.296725 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:54.296833 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:40631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260808 04:39:54.297272 15153 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.124:40631: Network error: Client connection negotiation failed: client connection to 127.15.228.124:40631: connect: Connection refused (error 111)
I20260808 04:39:54.298256 15264 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260808 04:39:54.298448 15264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-2/wal
uuid: "9c718b7c8b2048c085c92fe65d36354e"
format_stamp: "Formatted at 2026-08-08 04:39:54 on dist-test-slave-drl0"
server_key: "7a23f85398f4628d8312bf92c3daef8f"
server_key_iv: "7fa096998c9a91b71d14c00e9d1354d1"
server_key_version: "encryptionkey@0"
I20260808 04:39:54.298741 15264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:54.333690 15264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:54.339754 15264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:54.340003 15264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:54.355834 15264 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.124:40631
I20260808 04:39:54.356491 15351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.124:40631 every 8 connection(s)
I20260808 04:39:54.357920 15264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-2/data/info.pb
I20260808 04:39:54.363719 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 15264
I20260808 04:39:54.364005 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786163668922094-16273-0/minicluster-data/master-2/wal/instance
I20260808 04:39:54.364054 15353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:54.364537 16273 external_mini_cluster.cc:1428] Setting key 5009d279b2de48a7a93895b8e9f0c5a5
I20260808 04:39:54.365541 15153 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } attempt: 2
I20260808 04:39:54.365983 15258 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } attempt: 4
I20260808 04:39:54.376454 15353 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:39:54 dist-test-slave-drl0 krb5kdc[15043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163993, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for kudu/127.15.228.124@KRBTEST.COM
Aug 08 04:39:54 dist-test-slave-drl0 krb5kdc[15043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163993, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.125@KRBTEST.COM for kudu/127.15.228.124@KRBTEST.COM
Aug 08 04:39:54 dist-test-slave-drl0 krb5kdc[15043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.124@KRBTEST.COM
Aug 08 04:39:54 dist-test-slave-drl0 krb5kdc[15043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163994, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.124@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:39:54.397472 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.366238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:40631 (local address 127.0.0.1:46788)
0808 04:39:54.366564 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:54.366667 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:54.366821 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:54.372955 (+ 6134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:54.372969 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:54.373378 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:54.373617 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:54.373639 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.378379 (+ 4740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.378383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:54.379771 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:54.379784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.382168 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.382182 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:54.382401 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:54.383847 (+ 1446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:54.383891 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:54.386866 (+ 2975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:54.393125 (+ 6259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:54.393144 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:54.393150 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:54.393708 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:54.394539 (+ 831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:54.394544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:54.394546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:54.394746 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:54.396428 (+ 1682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:54.396445 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:54.396661 (+ 216us) client_negotiation.cc:769] Sending connection context
0808 04:39:54.396816 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 04:39:54.397092 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260808 04:39:54.398154 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.366746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:40631 (local address 127.0.0.1:46790)
0808 04:39:54.367215 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:54.367230 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:54.367519 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:54.372973 (+ 5454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:54.372978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:54.373357 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:54.373578 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:54.373588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.378498 (+ 4910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.378504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:54.379921 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:54.379946 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.382187 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.382205 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:54.382471 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:54.384124 (+ 1653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:54.384164 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:54.388167 (+ 4003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:54.394308 (+ 6141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:54.394315 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:54.394330 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:54.395116 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:54.395988 (+ 872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:54.396007 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:54.396009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:54.396212 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:54.397221 (+ 1009us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:54.397228 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:54.397417 (+ 189us) client_negotiation.cc:769] Sending connection context
0808 04:39:54.397576 (+ 159us) client_negotiation.cc:240] Negotiation successful
0808 04:39:54.397872 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326}
I20260808 04:39:54.400499 15360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.368151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46788 (local address 127.15.228.124:40631)
0808 04:39:54.368918 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.368966 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.369057 (+ 91us) server_negotiation.cc:407] Connection header received
0808 04:39:54.371366 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.371420 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.372302 (+ 882us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.373958 (+ 1656us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.375134 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.378084 (+ 2950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.380076 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.381920 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.383033 (+ 1113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:54.387464 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:54.387482 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:54.387495 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:54.387550 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:54.392774 (+ 5224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.393985 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.394009 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.394021 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.394302 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.395084 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.395091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.395096 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.396178 (+ 1082us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:54.398974 (+ 2796us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:54.399121 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 04:39:54.399954 (+ 833us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1836,"server-negotiator.queue_time_us":514,"spinlock_wait_cycles":88192,"thread_start_us":189,"threads_started":1}
I20260808 04:39:54.400491 15357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.368151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46776 (local address 127.15.228.124:40631)
0808 04:39:54.368922 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.368966 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.369057 (+ 91us) server_negotiation.cc:407] Connection header received
0808 04:39:54.371366 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.371420 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.372300 (+ 880us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.374600 (+ 2300us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.375135 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.378110 (+ 2975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.380076 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.381921 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.383034 (+ 1113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:54.390716 (+ 7682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:54.390738 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:54.390755 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:54.390828 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:54.395685 (+ 4857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.396971 (+ 1286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.396974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.396976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.397094 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.397686 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.397689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.397691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.398041 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:54.399299 (+ 1258us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:54.399387 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 04:39:54.399489 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1292,"server-negotiator.queue_time_us":656,"spinlock_wait_cycles":69120,"thread_start_us":188,"threads_started":1}
I20260808 04:39:54.400491 15361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.368791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46790 (local address 127.15.228.124:40631)
0808 04:39:54.369385 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.369410 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.369461 (+ 51us) server_negotiation.cc:407] Connection header received
0808 04:39:54.371407 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.371422 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.372295 (+ 873us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.373956 (+ 1661us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.375108 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.378198 (+ 3090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.380302 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.381907 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.383130 (+ 1223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:54.388626 (+ 5496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:54.388632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:54.388636 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:54.388704 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:54.394049 (+ 5345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.395490 (+ 1441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.395501 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.395506 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.395696 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.396655 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.396661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.396664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.397037 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:54.398471 (+ 1434us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:54.398771 (+ 300us) server_negotiation.cc:299] Negotiation successful
0808 04:39:54.399273 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":811,"server-negotiator.queue_time_us":449,"spinlock_wait_cycles":28160,"thread_start_us":134,"threads_started":1}
I20260808 04:39:54.407433 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.382532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.15.228.126:42957)
0808 04:39:54.384138 (+ 1606us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.384154 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.384986 (+ 832us) server_negotiation.cc:407] Connection header received
0808 04:39:54.385438 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.385456 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.385546 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.385725 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.387277 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.388403 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.391171 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.391798 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.392146 (+ 348us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:54.397474 (+ 5328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:54.397481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:54.397501 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:54.397551 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:54.401199 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.403828 (+ 2629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.403832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.403834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.404007 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.404774 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.404792 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.404794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.405296 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:54.405511 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:54.406758 (+ 1247us) server_negotiation.cc:299] Negotiation successful
0808 04:39:54.407049 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260808 04:39:54.408463 15362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.380855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:42957 (local address 127.0.0.1:36006)
0808 04:39:54.384760 (+ 3905us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:54.384864 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:54.385188 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:54.385842 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:54.385862 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:54.386692 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:54.387046 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:54.387056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.389226 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.389232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:54.390917 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:54.390979 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.392571 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.392576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:54.392787 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:54.393709 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:54.393757 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:54.397183 (+ 3426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:54.402972 (+ 5789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:54.402992 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:54.403000 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:54.403566 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:54.404248 (+ 682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:54.404251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:54.404252 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:54.404524 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:54.405641 (+ 1117us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:54.405660 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:54.406019 (+ 359us) client_negotiation.cc:769] Sending connection context
0808 04:39:54.407686 (+ 1667us) client_negotiation.cc:240] Negotiation successful
0808 04:39:54.408148 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3548,"thread_start_us":271,"threads_started":1}
I20260808 04:39:54.409368 15258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745: Bootstrap starting.
I20260808 04:39:54.409500 15153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b: Bootstrap starting.
I20260808 04:39:54.413004 15353 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:39:54.415446 15258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:54.415467 15153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:54.416884 15258 log.cc:824] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:54.417039 15153 log.cc:824] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:54.421167 15153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b: No bootstrap required, opened a new log
I20260808 04:39:54.421190 15258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745: No bootstrap required, opened a new log
Aug 08 04:39:54 dist-test-slave-drl0 krb5kdc[15043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
Aug 08 04:39:54 dist-test-slave-drl0 krb5kdc[15043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163994, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.124@KRBTEST.COM for kudu/127.15.228.125@KRBTEST.COM
I20260808 04:39:54.430621 15153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397576f71e4b4429a6fb93d48e30643b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } } peers { permanent_uuid: "c7fec59dcdf44b63ab3858550fa6e745" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } } peers { permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } }
I20260808 04:39:54.431363 15153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397576f71e4b4429a6fb93d48e30643b, State: Initialized, Role: FOLLOWER
I20260808 04:39:54.431365 15258 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397576f71e4b4429a6fb93d48e30643b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } } peers { permanent_uuid: "c7fec59dcdf44b63ab3858550fa6e745" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } } peers { permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } }
I20260808 04:39:54.431871 15258 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7fec59dcdf44b63ab3858550fa6e745, State: Initialized, Role: FOLLOWER
I20260808 04:39:54.432395 15153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397576f71e4b4429a6fb93d48e30643b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } } peers { permanent_uuid: "c7fec59dcdf44b63ab3858550fa6e745" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } } peers { permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } }
I20260808 04:39:54.432544 15258 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397576f71e4b4429a6fb93d48e30643b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } } peers { permanent_uuid: "c7fec59dcdf44b63ab3858550fa6e745" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } } peers { permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } }
I20260808 04:39:54.435616 15258 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:54.435643 15153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:54.435902 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.414628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.15.228.125:38973)
0808 04:39:54.414867 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.414872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.415332 (+ 460us) server_negotiation.cc:407] Connection header received
0808 04:39:54.415741 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.415746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.415870 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.415986 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.417281 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.418449 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.421627 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.422236 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.422378 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:54.429146 (+ 6768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:54.429161 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:54.429164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:54.429209 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:54.432262 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.433388 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.433392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.433394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.433526 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.434156 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.434160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.434161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.434614 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:54.434796 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:54.435424 (+ 628us) server_negotiation.cc:299] Negotiation successful
0808 04:39:54.435622 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260808 04:39:54.436677 15362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.414149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38973 (local address 127.0.0.1:59430)
0808 04:39:54.415180 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:54.415214 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:54.415463 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:54.416280 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:54.416284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:54.416572 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:54.416938 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:54.416949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.418764 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.418770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:54.421352 (+ 2582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:54.421388 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.422688 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.422706 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:54.423124 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:54.424084 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:54.424149 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:54.428814 (+ 4665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:54.432534 (+ 3720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:54.432548 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:54.432552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:54.433129 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:54.433775 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:54.433778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:54.433780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:54.433968 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:54.434895 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:54.434898 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:54.435128 (+ 230us) client_negotiation.cc:769] Sending connection context
0808 04:39:54.436161 (+ 1033us) client_negotiation.cc:240] Negotiation successful
0808 04:39:54.436412 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894}
I20260808 04:39:54.438050 15353 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:39:54.439592 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.412555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36018 (local address 127.15.228.126:42957)
0808 04:39:54.412781 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.412786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.412800 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:39:54.412910 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.412916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.413026 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.413536 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.415427 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.416461 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.418293 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.418772 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.419850 (+ 1078us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:54.424997 (+ 5147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:54.425003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:54.425006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:54.425240 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:54.429073 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.430583 (+ 1510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.430590 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.430603 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.430799 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.431969 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.431983 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.431984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.433226 (+ 1242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:54.433673 (+ 447us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:54.439128 (+ 5455us) server_negotiation.cc:299] Negotiation successful
0808 04:39:54.439357 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 04:39:54.442061 15365 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397576f71e4b4429a6fb93d48e30643b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } } peers { permanent_uuid: "c7fec59dcdf44b63ab3858550fa6e745" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } } peers { permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } } }
I20260808 04:39:54.442381 15365 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:39:54.441017 15369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397576f71e4b4429a6fb93d48e30643b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } } peers { permanent_uuid: "c7fec59dcdf44b63ab3858550fa6e745" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } } peers { permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } } }
I20260808 04:39:54.444237 15369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [sys.catalog]: This master's current role is: FOLLOWER
Aug 08 04:39:54 dist-test-slave-drl0 krb5kdc[15043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163994, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.124@KRBTEST.COM for kudu/127.15.228.124@KRBTEST.COM
Aug 08 04:39:54 dist-test-slave-drl0 krb5kdc[15043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.125@KRBTEST.COM
I20260808 04:39:54.452491 15164 master_runner.cc:186] Error getting master registration for 127.15.228.124:40631: OK, instance_id { permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" instance_seqno: 1786163994351137 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:39:54.460680 15164 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:39:54.453239 15060 master_runner.cc:186] Error getting master registration for 127.15.228.124:40631: OK, instance_id { permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" instance_seqno: 1786163994351137 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:39:54.462679 15060 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 04:39:54.462834 15396 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:39:54.463162 15396 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:39:54.463254 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.442762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59438 (local address 127.15.228.125:38973)
0808 04:39:54.442997 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.443003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.443018 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:39:54.443161 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.443163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.443243 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.443399 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.444931 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.445958 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.447737 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.448404 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.448619 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:54.454058 (+ 5439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:54.454074 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:54.454077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:54.454135 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:54.458310 (+ 4175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.459428 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.459432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.459434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.459594 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.460170 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.460174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.460188 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.460766 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:54.461151 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:54.462492 (+ 1341us) server_negotiation.cc:299] Negotiation successful
0808 04:39:54.462751 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 04:39:54.465968 15357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.439517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.15.228.124:40631)
0808 04:39:54.439727 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.439736 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.440064 (+ 328us) server_negotiation.cc:407] Connection header received
0808 04:39:54.440389 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.440402 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.440469 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.440587 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.442068 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.443245 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.445029 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.445563 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.445712 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:54.450550 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:54.450570 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:54.450587 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:54.450664 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:54.462375 (+ 11711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.463525 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.463531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.463536 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.463677 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.464151 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.464156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.464158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.464549 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:54.464731 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:54.465324 (+ 593us) server_negotiation.cc:299] Negotiation successful
0808 04:39:54.465557 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 04:39:54.466790 15362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.438919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:40631 (local address 127.0.0.1:46800)
0808 04:39:54.439903 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:54.439932 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:54.440189 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:54.441001 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:54.441005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:54.441522 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:54.441736 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:54.441757 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.443492 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.443495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:54.444808 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:54.444821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.445932 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.445947 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:54.446096 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:54.446981 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:54.447010 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:54.450264 (+ 3254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:54.462652 (+ 12388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:54.462671 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:54.462675 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:54.463273 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:54.463832 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:54.463848 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:54.463850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:54.463977 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:54.464799 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:54.464802 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:54.464978 (+ 176us) client_negotiation.cc:769] Sending connection context
0808 04:39:54.466214 (+ 1236us) client_negotiation.cc:240] Negotiation successful
0808 04:39:54.466498 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":175,"threads_started":1}
W20260808 04:39:54.468397 15401 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:39:54.468528 15401 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:39:54.472148 15353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e: Bootstrap starting.
I20260808 04:39:54.476125 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:39:54.477100 15353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:54.478838 15117 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "397576f71e4b4429a6fb93d48e30643b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36018
I20260808 04:39:54.479292 15353 log.cc:824] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:54.479481 15117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 04:39:54.479699 15117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [term 0 FOLLOWER]: Advancing to term 1
W20260808 04:39:54.482322 15089 tablet.cc:2369] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 04:39:54.482319 15117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397576f71e4b4429a6fb93d48e30643b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } } peers { permanent_uuid: "c7fec59dcdf44b63ab3858550fa6e745" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } } peers { permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } }
I20260808 04:39:54.486763 15353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e: No bootstrap required, opened a new log
I20260808 04:39:54.486676 15223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "397576f71e4b4429a6fb93d48e30643b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c7fec59dcdf44b63ab3858550fa6e745"
I20260808 04:39:54.487185 15223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:54.489447 15223 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 397576f71e4b4429a6fb93d48e30643b in term 1.
I20260808 04:39:54.490235 15086 leader_election.cc:302] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 397576f71e4b4429a6fb93d48e30643b, c7fec59dcdf44b63ab3858550fa6e745; no voters:
I20260808 04:39:54.490913 15369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:54.491477 15369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [term 1 LEADER]: Becoming Leader. State: Replica: 397576f71e4b4429a6fb93d48e30643b, State: Running, Role: LEADER
I20260808 04:39:54.492069 15369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397576f71e4b4429a6fb93d48e30643b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } } peers { permanent_uuid: "c7fec59dcdf44b63ab3858550fa6e745" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } } peers { permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } }
I20260808 04:39:54.495913 15353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397576f71e4b4429a6fb93d48e30643b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } } peers { permanent_uuid: "c7fec59dcdf44b63ab3858550fa6e745" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } } peers { permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } }
I20260808 04:39:54.496413 15353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c718b7c8b2048c085c92fe65d36354e, State: Initialized, Role: FOLLOWER
I20260808 04:39:54.497167 15353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397576f71e4b4429a6fb93d48e30643b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } } peers { permanent_uuid: "c7fec59dcdf44b63ab3858550fa6e745" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } } peers { permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } }
I20260808 04:39:54.500286 15353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:54.503903 15409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 397576f71e4b4429a6fb93d48e30643b. Latest consensus state: current_term: 1 leader_uuid: "397576f71e4b4429a6fb93d48e30643b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397576f71e4b4429a6fb93d48e30643b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } } peers { permanent_uuid: "c7fec59dcdf44b63ab3858550fa6e745" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } } peers { permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } } }
I20260808 04:39:54.504137 15409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:54.504120 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "397576f71e4b4429a6fb93d48e30643b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9c718b7c8b2048c085c92fe65d36354e"
I20260808 04:39:54.504532 15313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:54.505050 15424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:54.505260 15117 leader_election.cc:288] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [CANDIDATE]: Term 1 election: Requested vote from peers c7fec59dcdf44b63ab3858550fa6e745 (127.15.228.125:38973), 9c718b7c8b2048c085c92fe65d36354e (127.15.228.124:40631)
I20260808 04:39:54.507093 15313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 397576f71e4b4429a6fb93d48e30643b in term 1.
I20260808 04:39:54.508165 15424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:54.513731 15410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397576f71e4b4429a6fb93d48e30643b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } } peers { permanent_uuid: "c7fec59dcdf44b63ab3858550fa6e745" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } } peers { permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } } }
I20260808 04:39:54.513979 15410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:39:54.524163 15223 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745 [term 1 FOLLOWER]: Refusing update from remote peer 397576f71e4b4429a6fb93d48e30643b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:39:54.522502 15313 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e [term 1 FOLLOWER]: Refusing update from remote peer 397576f71e4b4429a6fb93d48e30643b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:39:54.525277 15409 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7fec59dcdf44b63ab3858550fa6e745" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:39:54.526345 15409 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:39:54.538714 15410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 397576f71e4b4429a6fb93d48e30643b. Latest consensus state: current_term: 1 leader_uuid: "397576f71e4b4429a6fb93d48e30643b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397576f71e4b4429a6fb93d48e30643b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } } peers { permanent_uuid: "c7fec59dcdf44b63ab3858550fa6e745" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } } peers { permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } } }
I20260808 04:39:54.539073 15410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e [sys.catalog]: This master's current role is: FOLLOWER
W20260808 04:39:54.545606 15439 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:39:54.545902 15439 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:39:54.549104 15365 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 397576f71e4b4429a6fb93d48e30643b. Latest consensus state: current_term: 1 leader_uuid: "397576f71e4b4429a6fb93d48e30643b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397576f71e4b4429a6fb93d48e30643b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } } peers { permanent_uuid: "c7fec59dcdf44b63ab3858550fa6e745" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } } peers { permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } } }
I20260808 04:39:54.549384 15365 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:39:54.556650 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.516415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.15.228.125:38973)
0808 04:39:54.516627 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.516631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.531492 (+ 14861us) server_negotiation.cc:407] Connection header received
0808 04:39:54.531753 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.531756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.531852 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.531995 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.533919 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.535209 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.537193 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.537687 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.537883 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:54.543177 (+ 5294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:54.543183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:54.543186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:54.543242 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:54.546217 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.547176 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.547181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.547183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.547337 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.547899 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.547903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.547905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.554774 (+ 6869us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:54.555049 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:54.556171 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0808 04:39:54.556377 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260808 04:39:54.561290 15409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "397576f71e4b4429a6fb93d48e30643b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397576f71e4b4429a6fb93d48e30643b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } } peers { permanent_uuid: "c7fec59dcdf44b63ab3858550fa6e745" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } } peers { permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } } }
I20260808 04:39:54.561515 15409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:54.562009 15434 mvcc.cc:205] Tried to move back new op lower bound from 7316127721551360000 to 7316127721445875712. Current Snapshot: MvccSnapshot[applied={T|T < 7316127721551360000}]
I20260808 04:39:54.564493 15410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "397576f71e4b4429a6fb93d48e30643b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397576f71e4b4429a6fb93d48e30643b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } } peers { permanent_uuid: "c7fec59dcdf44b63ab3858550fa6e745" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } } peers { permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } } }
I20260808 04:39:54.564718 15410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:39:54.564970 15442 mvcc.cc:205] Tried to move back new op lower bound from 7316127721551360000 to 7316127721445875712. Current Snapshot: MvccSnapshot[applied={T|T < 7316127721551360000}]
I20260808 04:39:54.568387 15369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "397576f71e4b4429a6fb93d48e30643b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397576f71e4b4429a6fb93d48e30643b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } } peers { permanent_uuid: "c7fec59dcdf44b63ab3858550fa6e745" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } } peers { permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } } }
I20260808 04:39:54.568625 15369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:54.570864 15365 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "397576f71e4b4429a6fb93d48e30643b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397576f71e4b4429a6fb93d48e30643b" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 42957 } } peers { permanent_uuid: "c7fec59dcdf44b63ab3858550fa6e745" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38973 } } peers { permanent_uuid: "9c718b7c8b2048c085c92fe65d36354e" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 40631 } } }
I20260808 04:39:54.571175 15365 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:39:54.581647 15357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.517546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46806 (local address 127.15.228.124:40631)
0808 04:39:54.517729 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.517733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.549809 (+ 32076us) server_negotiation.cc:407] Connection header received
0808 04:39:54.550033 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.550048 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.550128 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.550269 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.558959 (+ 8690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.560041 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.564770 (+ 4729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.565275 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.565422 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:54.572680 (+ 7258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:54.572684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:54.572687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:54.572718 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:54.575721 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.576576 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.576580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.576591 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.576733 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.578001 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.578005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.578007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.578416 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:54.578658 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:54.581129 (+ 2471us) server_negotiation.cc:299] Negotiation successful
0808 04:39:54.581371 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 04:39:54.592242 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:54.515619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36032 (local address 127.15.228.126:42957)
0808 04:39:54.515812 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.515817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.548929 (+ 33112us) server_negotiation.cc:407] Connection header received
0808 04:39:54.549229 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.549233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.549356 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.549515 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.560891 (+ 11376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.561944 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.566641 (+ 4697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:54.567165 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:54.567331 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:54.584544 (+ 17213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:54.584559 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:54.584563 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:54.584622 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:54.587921 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.589028 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.589033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.589034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.589192 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:54.590432 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:54.590436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:54.590439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:54.590834 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:54.591086 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:54.591744 (+ 658us) server_negotiation.cc:299] Negotiation successful
0808 04:39:54.591967 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 04:39:54.595333 15424 catalog_manager.cc:1347] Generated new cluster ID: 7289211e969f41b78a5a80682c326776
I20260808 04:39:54.595438 15424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:54.695637 15424 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:54.698246 15424 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:54.734663 15424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 397576f71e4b4429a6fb93d48e30643b: Generated new TSK 0
I20260808 04:39:54.736652 15424 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:39:54.810580 15263 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:54.804034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.15.228.125:38973)
0808 04:39:54.804237 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.804243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.807912 (+ 3669us) server_negotiation.cc:407] Connection header received
0808 04:39:54.808801 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.808805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.808898 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.809025 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.810359 (+ 1334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59470: BlockingRecv error: recv got EOF from 127.0.0.1:59470 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260808 04:39:54.813441 15160 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:54.805903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36034 (local address 127.15.228.126:42957)
0808 04:39:54.806120 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.806126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.811573 (+ 5447us) server_negotiation.cc:407] Connection header received
0808 04:39:54.811963 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.811969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.812067 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.812170 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.813207 (+ 1037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36034: BlockingRecv error: recv got EOF from 127.0.0.1:36034 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260808 04:39:54.816080 15357 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:54.807228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.15.228.124:40631)
0808 04:39:54.807464 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.807471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.814162 (+ 6691us) server_negotiation.cc:407] Connection header received
0808 04:39:54.814695 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.814702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.814791 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.814911 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.815852 (+ 941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46810: BlockingRecv error: recv got EOF from 127.0.0.1:46810 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260808 04:39:54.825675 15263 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:54.822592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.15.228.125:38973)
0808 04:39:54.822791 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.822808 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.823834 (+ 1026us) server_negotiation.cc:407] Connection header received
0808 04:39:54.824301 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.824305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.824468 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.824581 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.825516 (+ 935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59476: BlockingRecv error: recv got EOF from 127.0.0.1:59476 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260808 04:39:54.835974 15263 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:54.832343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.15.228.125:38973)
0808 04:39:54.832648 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.832654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.833829 (+ 1175us) server_negotiation.cc:407] Connection header received
0808 04:39:54.834268 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.834275 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.834383 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.834520 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.835779 (+ 1259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59492: BlockingRecv error: recv got EOF from 127.0.0.1:59492 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
W20260808 04:39:54.847110 15263 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:54.844262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59506 (local address 127.15.228.125:38973)
0808 04:39:54.844532 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.844538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.845133 (+ 595us) server_negotiation.cc:407] Connection header received
0808 04:39:54.845571 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.845575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.845691 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.845780 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.846868 (+ 1088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59506: BlockingRecv error: recv got EOF from 127.0.0.1:59506 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
W20260808 04:39:54.858866 15263 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:54.856095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.15.228.125:38973)
0808 04:39:54.856278 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.856293 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.857236 (+ 943us) server_negotiation.cc:407] Connection header received
0808 04:39:54.857602 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.857606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.857701 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.857792 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.858698 (+ 906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59514: BlockingRecv error: recv got EOF from 127.0.0.1:59514 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260808 04:39:54.874979 15263 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:54.871397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59526 (local address 127.15.228.125:38973)
0808 04:39:54.871617 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.871622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.872852 (+ 1230us) server_negotiation.cc:407] Connection header received
0808 04:39:54.873281 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.873295 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.873402 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.873550 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.874721 (+ 1171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59526: BlockingRecv error: recv got EOF from 127.0.0.1:59526 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260808 04:39:54.898576 15263 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:54.895680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59528 (local address 127.15.228.125:38973)
0808 04:39:54.895936 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.895953 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.896827 (+ 874us) server_negotiation.cc:407] Connection header received
0808 04:39:54.897117 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.897121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.897211 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.897354 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.898368 (+ 1014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59528: BlockingRecv error: recv got EOF from 127.0.0.1:59528 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260808 04:39:54.940371 15263 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:54.938447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59542 (local address 127.15.228.125:38973)
0808 04:39:54.938710 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:54.938715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:54.938730 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:39:54.938831 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:54.938835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:54.938969 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:54.939108 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:54.940078 (+ 970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59542: BlockingRecv error: recv got EOF from 127.0.0.1:59542 (error 108)
Metrics: {"server-negotiator.queue_time_us":153}
W20260808 04:39:55.012344 15263 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:55.010184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59552 (local address 127.15.228.125:38973)
0808 04:39:55.010437 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:55.010441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:55.010847 (+ 406us) server_negotiation.cc:407] Connection header received
0808 04:39:55.011093 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:55.011097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:55.011183 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:55.011324 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:55.012198 (+ 874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59552: BlockingRecv error: recv got EOF from 127.0.0.1:59552 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
W20260808 04:39:55.147495 15263 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:55.144780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59566 (local address 127.15.228.125:38973)
0808 04:39:55.145099 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:55.145106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:55.145443 (+ 337us) server_negotiation.cc:407] Connection header received
0808 04:39:55.145718 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:55.145722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:55.145814 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:55.145977 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:55.147074 (+ 1097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59566: BlockingRecv error: recv got EOF from 127.0.0.1:59566 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
W20260808 04:39:55.408313 15263 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:55.406786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.15.228.125:38973)
0808 04:39:55.407079 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:55.407084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:55.407232 (+ 148us) server_negotiation.cc:407] Connection header received
0808 04:39:55.407313 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:55.407316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:55.407370 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:55.407451 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:55.408179 (+ 728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59580: BlockingRecv error: recv got EOF from 127.0.0.1:59580 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
I20260808 04:39:55.466228 15396 catalog_manager.cc:1259] Loaded cluster ID: 7289211e969f41b78a5a80682c326776
I20260808 04:39:55.466360 15396 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745: loading cluster ID for follower catalog manager: success
I20260808 04:39:55.468544 15396 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745: acquiring CA information for follower catalog manager: success
I20260808 04:39:55.470515 15396 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c7fec59dcdf44b63ab3858550fa6e745: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:39:55.547365 15439 catalog_manager.cc:1259] Loaded cluster ID: 7289211e969f41b78a5a80682c326776
I20260808 04:39:55.547464 15439 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e: loading cluster ID for follower catalog manager: success
I20260808 04:39:55.549458 15439 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e: acquiring CA information for follower catalog manager: success
I20260808 04:39:55.550787 15439 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9c718b7c8b2048c085c92fe65d36354e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:39:55.927973 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:55.922903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.15.228.125:38973)
0808 04:39:55.923220 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:55.923224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:55.923302 (+ 78us) server_negotiation.cc:407] Connection header received
0808 04:39:55.923553 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:55.923556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:55.923708 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:55.923829 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:39:55.924230 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:55.924795 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:55.925804 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:55.926169 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:55.926283 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:55.926642 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:39:55.927253 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:39:55.927376 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:55.927527 (+ 151us) server_negotiation.cc:299] Negotiation successful
0808 04:39:55.927769 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":91,"threads_started":1}
I20260808 04:39:55.936878 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:55.932207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46820 (local address 127.15.228.124:40631)
0808 04:39:55.932625 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:55.932629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:55.932643 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:39:55.932730 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:55.932732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:55.932851 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:55.932959 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:39:55.933261 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:55.933794 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:55.934691 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:55.934979 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:55.935074 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:55.935697 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:39:55.936349 (+ 652us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:39:55.936525 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:55.936619 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 04:39:55.936685 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":121,"threads_started":1}
I20260808 04:39:55.945958 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:55.940540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59612 (local address 127.15.228.125:38973)
0808 04:39:55.941125 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:55.941130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:55.941195 (+ 65us) server_negotiation.cc:407] Connection header received
0808 04:39:55.941381 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:55.941384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:55.941440 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:55.941534 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:39:55.942157 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:55.943031 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:55.944756 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:55.945020 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:55.945143 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:55.945410 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:39:55.945501 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:39:55.945579 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:55.945737 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 04:39:55.945802 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486}
I20260808 04:39:55.946435 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:55.941029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46832 (local address 127.15.228.124:40631)
0808 04:39:55.941144 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:55.941150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:55.941352 (+ 202us) server_negotiation.cc:407] Connection header received
0808 04:39:55.941534 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:55.941538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:55.941586 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:55.941662 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:39:55.942154 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:55.943177 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:55.945256 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:55.945588 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:55.945722 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:55.945988 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:39:55.946073 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:39:55.946152 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:55.946260 (+ 108us) server_negotiation.cc:299] Negotiation successful
0808 04:39:55.946315 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 04:39:55.947063 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:55.940549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.15.228.126:42957)
0808 04:39:55.940824 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:55.940830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:55.941096 (+ 266us) server_negotiation.cc:407] Connection header received
0808 04:39:55.941218 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:55.941221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:55.941413 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:55.941547 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:39:55.942156 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:55.943200 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:55.945251 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:55.945649 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:55.945750 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:55.946190 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:39:55.946569 (+ 379us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:39:55.946666 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:55.946820 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 04:39:55.946889 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":110,"threads_started":1}
I20260808 04:39:55.951347 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 15060
I20260808 04:39:55.971555 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 15164
I20260808 04:39:55.984661 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 15264
2026-08-08T04:39:55Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4468 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 04:39:56 dist-test-slave-drl0 krb5kdc[15558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 04:39:56 dist-test-slave-drl0 krb5kdc[15558](info): set up 2 sockets
Aug 08 04:39:56 dist-test-slave-drl0 krb5kdc[15558](info): commencing operation
krb5kdc: starting...
W20260808 04:39:57.283995 16273 mini_kdc.cc:121] Time spent starting KDC: real 1.269s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/test-user.keytab.
Aug 08 04:39:57 dist-test-slave-drl0 krb5kdc[15558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T04:39:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T04:39:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.228.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
Entry for principal kudu/127.15.228.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab.
I20260808 04:39:57.432209 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:36939
--webserver_interface=127.15.228.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:42785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:36939,127.15.228.125:38507,127.15.228.124:36725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 04:39:57.569769 15573 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:39:57.570115 15573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:57.570186 15573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:57.570264 15573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:57.574338 15573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:57.574436 15573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:57.574474 15573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:57.574512 15573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:57.574548 15573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:57.582139 15573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:42785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:36939,127.15.228.125:38507,127.15.228.124:36725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.126.keytab
--principal=kudu/127.15.228.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.126:36939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.228.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:57.584460 15573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:57.586344 15573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:57.596572 15578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:57.596504 15583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:57.596771 15573 server_base.cc:1034] running on GCE node
W20260808 04:39:57.596504 15580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:57.597472 15573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:57.598790 15573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:57.600286 15573 hybrid_clock.cc:648] HybridClock initialized: now 1786163997600253 us; error 41 us; skew 500 ppm
Aug 08 04:39:57 dist-test-slave-drl0 krb5kdc[15558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163997, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:57.604889 15573 init.cc:377] Logged in from keytab as kudu/127.15.228.126@KRBTEST.COM (short username kudu)
I20260808 04:39:57.606483 15573 webserver.cc:460] Webserver started at http://127.15.228.126:44271/ using document root <none> and password file <none>
I20260808 04:39:57.607246 15573 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:57.607331 15573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:57.607542 15573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:57.609773 15573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-0/data/instance:
uuid: "e3214d734db7468092d8e01637f3efa8"
format_stamp: "Formatted at 2026-08-08 04:39:57 on dist-test-slave-drl0"
server_key: "06cb8461f86eda71a02710db2ce4a13c"
server_key_iv: "93136ca4672b26c358770f50e9be7e17"
server_key_version: "encryptionkey@0"
I20260808 04:39:57.610337 15573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-0/wal/instance:
uuid: "e3214d734db7468092d8e01637f3efa8"
format_stamp: "Formatted at 2026-08-08 04:39:57 on dist-test-slave-drl0"
server_key: "06cb8461f86eda71a02710db2ce4a13c"
server_key_iv: "93136ca4672b26c358770f50e9be7e17"
server_key_version: "encryptionkey@0"
I20260808 04:39:57.615221 15573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 04:39:57.618533 15593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:57.620815 15573 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 04:39:57.621261 15573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-0/wal
uuid: "e3214d734db7468092d8e01637f3efa8"
format_stamp: "Formatted at 2026-08-08 04:39:57 on dist-test-slave-drl0"
server_key: "06cb8461f86eda71a02710db2ce4a13c"
server_key_iv: "93136ca4672b26c358770f50e9be7e17"
server_key_version: "encryptionkey@0"
I20260808 04:39:57.621485 15573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:57.635748 15573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:57.640406 15573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:57.640702 15573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:57.649834 15573 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.126:36939
I20260808 04:39:57.650148 15665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.126:36939 every 8 connection(s)
I20260808 04:39:57.651054 15573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-0/data/info.pb
I20260808 04:39:57.652189 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 15573
I20260808 04:39:57.652294 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-0/wal/instance
I20260808 04:39:57.652622 16273 external_mini_cluster.cc:1428] Setting key 2ce1ae4bd244f05b8a0d3af106ce8b16
I20260808 04:39:57.654238 15666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:57.658710 15666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.228.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.125@KRBTEST.COM" created.
Aug 08 04:39:57 dist-test-slave-drl0 krb5kdc[15558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163997, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:39:57.678433 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:57.661221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55442 (local address 127.15.228.126:36939)
0808 04:39:57.661798 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:57.661816 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:57.662026 (+ 210us) server_negotiation.cc:407] Connection header received
0808 04:39:57.663982 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:57.663987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:57.664175 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:57.664349 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:57.665830 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:57.667361 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:57.668592 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:57.668980 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:57.669218 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:57.672298 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:57.672304 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:57.672314 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:57.672344 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:57.674162 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:57.674805 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:57.674811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:57.674812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:57.675029 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:57.675446 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:57.675449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:57.675451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:57.675989 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:57.676113 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:57.677134 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0808 04:39:57.677264 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"spinlock_wait_cycles":16768,"thread_start_us":167,"threads_started":1}
I20260808 04:39:57.678433 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:57.661221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36939 (local address 127.0.0.1:55442)
0808 04:39:57.661822 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:57.661891 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:57.663559 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:57.664387 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:57.664391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:57.665006 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:57.665326 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:57.665343 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:57.667548 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:57.667554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:57.668460 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:57.668471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:57.669224 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:57.669228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:57.669326 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:57.669988 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:57.670005 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:57.672081 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:57.674288 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:57.674301 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:57.674314 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:57.674645 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:57.675141 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:57.675144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:57.675146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:57.675309 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:57.676146 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:57.676156 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:57.676364 (+ 208us) client_negotiation.cc:769] Sending connection context
0808 04:39:57.676587 (+ 223us) client_negotiation.cc:240] Negotiation successful
0808 04:39:57.676966 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":148,"threads_started":1}
I20260808 04:39:57.681457 15666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:39:57.682484 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:57.682150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38507 (local address 127.0.0.1:36574)
0808 04:39:57.682324 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:57.682406 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 04:39:57.683266 15597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38507: connect: Connection refused (error 111)
I20260808 04:39:57.684530 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:57.684300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38507 (local address 127.0.0.1:36578)
0808 04:39:57.684423 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:57.684470 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 04:39:57.685024 15666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.125:38507: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38507: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.125.keytab.
Entry for principal kudu/127.15.228.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.125.keytab.
I20260808 04:39:57.687680 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.125:38507
--webserver_interface=127.15.228.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.125.keytab
--principal=kudu/127.15.228.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:42785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:36939,127.15.228.125:38507,127.15.228.124:36725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:39:57.734395 15666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } attempt: 1
I20260808 04:39:57.735756 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:57.735361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38507 (local address 127.0.0.1:36580)
0808 04:39:57.735590 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:57.735661 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260808 04:39:57.736644 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:57.736380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38507 (local address 127.0.0.1:36590)
0808 04:39:57.736529 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:57.736585 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260808 04:39:57.736826 15666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.125:38507: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38507: connect: Connection refused (error 111)
I20260808 04:39:57.805022 15666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } attempt: 2
I20260808 04:39:57.806185 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:57.805852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38507 (local address 127.0.0.1:36596)
0808 04:39:57.806042 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:57.806107 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 04:39:57.807163 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:57.806962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38507 (local address 127.0.0.1:36608)
0808 04:39:57.807061 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:57.807108 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260808 04:39:57.807327 15666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.125:38507: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38507: connect: Connection refused (error 111)
W20260808 04:39:57.821810 15679 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:39:57.822396 15679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:57.822468 15679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:57.822695 15679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:57.828231 15679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:57.828332 15679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:57.828373 15679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:57.828444 15679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:57.828496 15679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:57.834252 15679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:42785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:36939,127.15.228.125:38507,127.15.228.124:36725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.125.keytab
--principal=kudu/127.15.228.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.125:38507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.228.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:57.835575 15679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:57.836807 15679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:57.845866 15687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:57.845901 15689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:57.846021 15684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:57.846204 15679 server_base.cc:1034] running on GCE node
I20260808 04:39:57.846824 15679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:57.848022 15679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:57.849390 15679 hybrid_clock.cc:648] HybridClock initialized: now 1786163997849318 us; error 71 us; skew 500 ppm
Aug 08 04:39:57 dist-test-slave-drl0 krb5kdc[15558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163997, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:57.852466 15679 init.cc:377] Logged in from keytab as kudu/127.15.228.125@KRBTEST.COM (short username kudu)
I20260808 04:39:57.853700 15679 webserver.cc:460] Webserver started at http://127.15.228.125:44605/ using document root <none> and password file <none>
I20260808 04:39:57.854315 15679 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:57.854388 15679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:57.854614 15679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:57.856915 15679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-1/data/instance:
uuid: "0c7d1b7607234fea8bbc5c770d3eee4e"
format_stamp: "Formatted at 2026-08-08 04:39:57 on dist-test-slave-drl0"
server_key: "b20ab74dc04976a7c30b3751283acd8d"
server_key_iv: "19a07710e0375a4006088d7bb4adaf6f"
server_key_version: "encryptionkey@0"
I20260808 04:39:57.857537 15679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-1/wal/instance:
uuid: "0c7d1b7607234fea8bbc5c770d3eee4e"
format_stamp: "Formatted at 2026-08-08 04:39:57 on dist-test-slave-drl0"
server_key: "b20ab74dc04976a7c30b3751283acd8d"
server_key_iv: "19a07710e0375a4006088d7bb4adaf6f"
server_key_version: "encryptionkey@0"
I20260808 04:39:57.861725 15679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260808 04:39:57.864795 15697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:57.866137 15679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 04:39:57.866271 15679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-1/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-1/wal
uuid: "0c7d1b7607234fea8bbc5c770d3eee4e"
format_stamp: "Formatted at 2026-08-08 04:39:57 on dist-test-slave-drl0"
server_key: "b20ab74dc04976a7c30b3751283acd8d"
server_key_iv: "19a07710e0375a4006088d7bb4adaf6f"
server_key_version: "encryptionkey@0"
I20260808 04:39:57.866425 15679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:57.888033 15679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:57.891577 15679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:57.891752 15679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:57.898515 15666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } attempt: 3
I20260808 04:39:57.899685 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:57.899336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38507 (local address 127.0.0.1:36612)
0808 04:39:57.899527 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:57.899597 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260808 04:39:57.900462 15679 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.125:38507
I20260808 04:39:57.900530 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:57.900269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38507 (local address 127.0.0.1:36628)
0808 04:39:57.900396 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:57.900455 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 04:39:57.900682 15666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.125:38507: Network error: Client connection negotiation failed: client connection to 127.15.228.125:38507: connect: Connection refused (error 111)
I20260808 04:39:57.900823 15772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.125:38507 every 8 connection(s)
I20260808 04:39:57.901597 15679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-1/data/info.pb
I20260808 04:39:57.904613 15774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 04:39:57.907382 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 15679
I20260808 04:39:57.907639 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-1/wal/instance
I20260808 04:39:57.907925 16273 external_mini_cluster.cc:1428] Setting key 98209d67ea635c8de9211d7b0210e7a7
I20260808 04:39:57.908787 15774 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.228.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.228.124@KRBTEST.COM" created.
Aug 08 04:39:57 dist-test-slave-drl0 krb5kdc[15558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163997, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.125@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:39:57.927070 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:57.911004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55450 (local address 127.15.228.126:36939)
0808 04:39:57.911171 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:57.911176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:57.912266 (+ 1090us) server_negotiation.cc:407] Connection header received
0808 04:39:57.913297 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:57.913303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:57.913372 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:57.913485 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:57.914776 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:57.915785 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:57.917203 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:57.917638 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:57.917785 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:57.922034 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:57.922038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:57.922040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:57.922067 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:57.923592 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:57.924291 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:57.924299 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:57.924300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:57.924496 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:57.925146 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:57.925151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:57.925153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:57.925412 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:57.925542 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:57.926431 (+ 889us) server_negotiation.cc:299] Negotiation successful
0808 04:39:57.926800 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:39:57.928318 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:57.911458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36939 (local address 127.0.0.1:55450)
0808 04:39:57.912072 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:57.912130 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:57.913074 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:57.913658 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:57.913664 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:57.914014 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:57.914605 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:57.914618 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:57.916032 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:57.916035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:57.917064 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:57.917076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:57.917947 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:57.918166 (+ 219us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:57.918449 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:57.919683 (+ 1234us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:57.919702 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:57.921825 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:57.923737 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:57.923744 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:57.923763 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:57.924049 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:57.924723 (+ 674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:57.924728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:57.924731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:57.924997 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:57.925540 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:57.925550 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:57.926021 (+ 471us) client_negotiation.cc:769] Sending connection context
0808 04:39:57.926344 (+ 323us) client_negotiation.cc:240] Negotiation successful
0808 04:39:57.926980 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":259,"threads_started":1}
I20260808 04:39:57.930604 15774 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 04:39:57 dist-test-slave-drl0 krb5kdc[15558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163997, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.125@KRBTEST.COM for kudu/127.15.228.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.228.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.124.keytab.
Entry for principal kudu/127.15.228.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.124.keytab.
I20260808 04:39:57.939268 16273 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
/tmp/dist-test-taskhFaBKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.124:36725
--webserver_interface=127.15.228.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.124.keytab
--principal=kudu/127.15.228.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.228.84:42785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.228.126:36939,127.15.228.125:38507,127.15.228.124:36725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 04:39:57.942970 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:57.931460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38507 (local address 127.0.0.1:36642)
0808 04:39:57.931661 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:57.931675 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:57.931766 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:57.932595 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:57.932598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:57.932805 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:57.932902 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:57.932909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:57.934352 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:57.934359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:57.935317 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:57.935328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:57.936013 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:57.936016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:57.936120 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:57.936708 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:57.936720 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:57.938135 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:57.940965 (+ 2830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:57.940968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:57.940971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:57.941180 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:57.941661 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:57.941664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:57.941666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:57.941764 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:57.942313 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:57.942315 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:57.942404 (+ 89us) client_negotiation.cc:769] Sending connection context
0808 04:39:57.942530 (+ 126us) client_negotiation.cc:240] Negotiation successful
0808 04:39:57.942678 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":1}
I20260808 04:39:57.942983 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:57.931733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36642 (local address 127.15.228.125:38507)
0808 04:39:57.932152 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:57.932163 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:57.932194 (+ 31us) server_negotiation.cc:407] Connection header received
0808 04:39:57.932232 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:57.932236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:57.932440 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:57.932603 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:57.933029 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:57.933969 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:57.935462 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:57.935884 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:57.936013 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:57.938279 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:57.938289 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:57.938301 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:57.938322 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:57.940794 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:57.941312 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:57.941324 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:57.941325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:57.941413 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:57.941870 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:57.941883 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:57.941885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:57.942186 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:57.942364 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:57.942619 (+ 255us) server_negotiation.cc:299] Negotiation successful
0808 04:39:57.942755 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":180,"threads_started":1}
I20260808 04:39:57.945087 15774 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:39:57.945983 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:57.945675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36725 (local address 127.0.0.1:46434)
0808 04:39:57.945829 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:57.945919 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260808 04:39:57.946204 15698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.228.124:36725: connect: Connection refused (error 111)
I20260808 04:39:57.947124 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:57.946880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36725 (local address 127.0.0.1:46448)
0808 04:39:57.947030 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:57.947071 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260808 04:39:57.947307 15774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.124:36725: Network error: Client connection negotiation failed: client connection to 127.15.228.124:36725: connect: Connection refused (error 111)
I20260808 04:39:57.996589 15774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } attempt: 1
I20260808 04:39:57.997997 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:57.997620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36725 (local address 127.0.0.1:46450)
0808 04:39:57.997808 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:57.997886 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 04:39:57.999128 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:57.998827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36725 (local address 127.0.0.1:46458)
0808 04:39:57.999002 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:57.999062 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260808 04:39:57.999296 15774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.124:36725: Network error: Client connection negotiation failed: client connection to 127.15.228.124:36725: connect: Connection refused (error 111)
I20260808 04:39:58.044016 15666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } attempt: 4
Aug 08 04:39:58 dist-test-slave-drl0 krb5kdc[15558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163997, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for kudu/127.15.228.125@KRBTEST.COM
I20260808 04:39:58.056016 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.044953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38507 (local address 127.0.0.1:36652)
0808 04:39:58.045350 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:58.045365 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:58.045550 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:58.045836 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:58.045839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:58.046055 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:58.046175 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:58.046180 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.047594 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.047600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:58.048554 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:58.048563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.049097 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.049101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:58.049213 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:58.049775 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:58.049789 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:58.051685 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:58.054077 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:58.054082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:58.054087 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:58.054408 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:58.054754 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:58.054758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:58.054760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:58.054880 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:58.055364 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:58.055368 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:58.055496 (+ 128us) client_negotiation.cc:769] Sending connection context
0808 04:39:58.055629 (+ 133us) client_negotiation.cc:240] Negotiation successful
0808 04:39:58.055804 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285}
I20260808 04:39:58.056047 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.045103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36652 (local address 127.15.228.125:38507)
0808 04:39:58.045272 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.045277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.045470 (+ 193us) server_negotiation.cc:407] Connection header received
0808 04:39:58.045660 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.045663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.045719 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.045816 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.046298 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.047454 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.048687 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.048954 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.049076 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:58.051965 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:58.051970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:58.051972 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:58.052009 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:58.053899 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.054554 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.054557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.054558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.054642 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.055034 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.055038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.055039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.055248 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:58.055339 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:58.055777 (+ 438us) server_negotiation.cc:299] Negotiation successful
0808 04:39:58.055922 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 04:39:58.057154 15666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:39:58.057997 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.057717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36725 (local address 127.0.0.1:46474)
0808 04:39:58.057885 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:58.057936 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260808 04:39:58.058704 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.058497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36725 (local address 127.0.0.1:46482)
0808 04:39:58.058606 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:58.058657 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 04:39:58.058876 15666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.124:36725: Network error: Client connection negotiation failed: client connection to 127.15.228.124:36725: connect: Connection refused (error 111)
I20260808 04:39:58.067514 15774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } attempt: 2
I20260808 04:39:58.068822 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.068364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36725 (local address 127.0.0.1:46492)
0808 04:39:58.068580 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:58.068699 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260808 04:39:58.070053 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.069699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36725 (local address 127.0.0.1:46498)
0808 04:39:58.069887 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:58.069961 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260808 04:39:58.070303 15774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.124:36725: Network error: Client connection negotiation failed: client connection to 127.15.228.124:36725: connect: Connection refused (error 111)
W20260808 04:39:58.084398 15784 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 04:39:58.084753 15784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 04:39:58.084779 15784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 04:39:58.084869 15784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 04:39:58.090999 15784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 04:39:58.091084 15784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 04:39:58.091104 15784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 04:39:58.091123 15784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 04:39:58.091140 15784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 04:39:58.098169 15784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.228.84:42785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.228.126:36939,127.15.228.125:38507,127.15.228.124:36725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/krb5kdc/kudu_127.15.228.124.keytab
--principal=kudu/127.15.228.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.228.124:36725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.228.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 08 Aug 2026 03:51:01 UTC on ccycloud.quasar-lceqdi.root.comops.site
build id 4574013
I20260808 04:39:58.099462 15784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 04:39:58.100564 15784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 04:39:58.109782 15799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:58.109781 15797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 04:39:58.109808 15794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 04:39:58.110138 15784 server_base.cc:1034] running on GCE node
I20260808 04:39:58.110874 15784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 04:39:58.112108 15784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 04:39:58.115449 15784 hybrid_clock.cc:648] HybridClock initialized: now 1786163998115452 us; error 112 us; skew 500 ppm
I20260808 04:39:58.118124 15666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } attempt: 1
Aug 08 04:39:58 dist-test-slave-drl0 krb5kdc[15558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163998, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 04:39:58.118821 15784 init.cc:377] Logged in from keytab as kudu/127.15.228.124@KRBTEST.COM (short username kudu)
I20260808 04:39:58.119429 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.118918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36725 (local address 127.0.0.1:46512)
0808 04:39:58.119228 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:58.119316 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 04:39:58.120069 15784 webserver.cc:460] Webserver started at http://127.15.228.124:38465/ using document root <none> and password file <none>
I20260808 04:39:58.120442 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.120144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36725 (local address 127.0.0.1:46516)
0808 04:39:58.120306 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:58.120366 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.228.124:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260808 04:39:58.120628 15666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.228.124:36725: Network error: Client connection negotiation failed: client connection to 127.15.228.124:36725: connect: Connection refused (error 111)
I20260808 04:39:58.120836 15784 fs_manager.cc:359] Metadata directory not provided
I20260808 04:39:58.120936 15784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 04:39:58.121218 15784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 04:39:58.123524 15784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-2/data/instance:
uuid: "03ee39c597fe4dc5bec573b11c3eabf5"
format_stamp: "Formatted at 2026-08-08 04:39:58 on dist-test-slave-drl0"
server_key: "786c62c1b8031ac8c69c347633364a26"
server_key_iv: "3815e4490d8264ce08773afe0ac0623f"
server_key_version: "encryptionkey@0"
I20260808 04:39:58.124173 15784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-2/wal/instance:
uuid: "03ee39c597fe4dc5bec573b11c3eabf5"
format_stamp: "Formatted at 2026-08-08 04:39:58 on dist-test-slave-drl0"
server_key: "786c62c1b8031ac8c69c347633364a26"
server_key_iv: "3815e4490d8264ce08773afe0ac0623f"
server_key_version: "encryptionkey@0"
I20260808 04:39:58.128535 15784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:39:58.131937 15810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 04:39:58.133718 15784 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 04:39:58.133976 15784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-2/data,/tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-2/wal
uuid: "03ee39c597fe4dc5bec573b11c3eabf5"
format_stamp: "Formatted at 2026-08-08 04:39:58 on dist-test-slave-drl0"
server_key: "786c62c1b8031ac8c69c347633364a26"
server_key_iv: "3815e4490d8264ce08773afe0ac0623f"
server_key_version: "encryptionkey@0"
I20260808 04:39:58.134177 15784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 04:39:58.149952 15784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 04:39:58.153599 15784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 04:39:58.153828 15784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 04:39:58.161523 15774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } attempt: 3
I20260808 04:39:58.162056 15784 rpc_server.cc:307] RPC server started. Bound to: 127.15.228.124:36725
I20260808 04:39:58.162357 15875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.228.124:36725 every 8 connection(s)
I20260808 04:39:58.163327 15784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-2/data/info.pb
I20260808 04:39:58.166429 15876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 04:39:58 dist-test-slave-drl0 krb5kdc[15558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163997, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.125@KRBTEST.COM for kudu/127.15.228.124@KRBTEST.COM
I20260808 04:39:58.171576 16273 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu as pid 15784
I20260808 04:39:58.171700 16273 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFaBKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786163668922094-16273-0/minicluster-data/master-2/wal/instance
I20260808 04:39:58.171489 15876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:39:58.171967 16273 external_mini_cluster.cc:1428] Setting key 524648eb922930e2ecb61e5c191c600c
I20260808 04:39:58.176136 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.162282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36725 (local address 127.0.0.1:46526)
0808 04:39:58.162448 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:58.162467 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:58.162568 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:58.165414 (+ 2846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:58.165425 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:58.165635 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:58.165759 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:58.165767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.167299 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.167302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:58.168090 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:58.168104 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.168853 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.168857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:58.168945 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:58.169710 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:58.169728 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:58.171412 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:58.173802 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:58.173818 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:58.173821 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:58.174099 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:58.174569 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:58.174574 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:58.174576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:58.174687 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:58.175528 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:58.175533 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:58.175662 (+ 129us) client_negotiation.cc:769] Sending connection context
0808 04:39:58.175792 (+ 130us) client_negotiation.cc:240] Negotiation successful
0808 04:39:58.175951 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 04:39:58.177299 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.163468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46526 (local address 127.15.228.124:36725)
0808 04:39:58.164016 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.164043 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.164083 (+ 40us) server_negotiation.cc:407] Connection header received
0808 04:39:58.164753 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.164769 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.165119 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.165418 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.165958 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.167166 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.168250 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.168711 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.168847 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:58.171619 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:58.171625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:58.171638 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:58.171666 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:58.173659 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.174261 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.174270 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.174281 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.174425 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.174816 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.174821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.174825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.175395 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:58.175531 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:58.175952 (+ 421us) server_negotiation.cc:299] Negotiation successful
0808 04:39:58.176243 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":328,"thread_start_us":178,"threads_started":1}
Aug 08 04:39:58 dist-test-slave-drl0 krb5kdc[15558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.124@KRBTEST.COM
Aug 08 04:39:58 dist-test-slave-drl0 krb5kdc[15558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163998, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.124@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:39:58.182535 15774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e: Bootstrap starting.
I20260808 04:39:58.184940 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.174428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55462 (local address 127.15.228.126:36939)
0808 04:39:58.174632 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.174638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.174652 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:39:58.174697 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.174700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.174759 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.174842 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.175618 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.176399 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.177434 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.177787 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.177886 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:58.181045 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:58.181049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:58.181052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:58.181087 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:58.183048 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.183536 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.183539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.183541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.183631 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.183925 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.183928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.183930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.184199 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:58.184300 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:58.184691 (+ 391us) server_negotiation.cc:299] Negotiation successful
0808 04:39:58.184811 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260808 04:39:58.185562 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.173665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.126:36939 (local address 127.0.0.1:55462)
0808 04:39:58.173993 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:58.174041 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:58.174143 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:58.174878 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:58.174892 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:58.175259 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:58.175435 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:58.175442 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.176530 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.176533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:58.177296 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:58.177307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.178057 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.178061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:58.178206 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:58.178697 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:58.178716 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:58.180826 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:58.183156 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:58.183166 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:58.183173 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:58.183413 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:58.183728 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:58.183731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:58.183733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:58.183822 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:58.184359 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:58.184365 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:58.184457 (+ 92us) client_negotiation.cc:769] Sending connection context
0808 04:39:58.185127 (+ 670us) client_negotiation.cc:240] Negotiation successful
0808 04:39:58.185334 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":35,"thread_start_us":96,"threads_started":1}
I20260808 04:39:58.185580 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.173127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46540 (local address 127.15.228.124:36725)
0808 04:39:58.173445 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.173453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.173469 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:39:58.173510 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.173515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.173571 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.173665 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.174874 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.175655 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.176831 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.177182 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.177296 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:58.180570 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:58.180574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:58.180577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:58.180604 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:58.182802 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.183340 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.183344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.183345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.183445 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.184219 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.184225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.184229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.184456 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:58.184638 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:58.185201 (+ 563us) server_negotiation.cc:299] Negotiation successful
0808 04:39:58.185398 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":247,"thread_start_us":131,"threads_started":1}
I20260808 04:39:58.186633 15876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:39:58.186704 15774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:58.187758 15774 log.cc:824] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:58.187808 15666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } attempt: 2
I20260808 04:39:58.190327 15774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e: No bootstrap required, opened a new log
Aug 08 04:39:58 dist-test-slave-drl0 krb5kdc[15558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163998, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.124@KRBTEST.COM for kudu/127.15.228.125@KRBTEST.COM
Aug 08 04:39:58 dist-test-slave-drl0 krb5kdc[15558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163997, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.126@KRBTEST.COM for kudu/127.15.228.124@KRBTEST.COM
I20260808 04:39:58.195806 15774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3214d734db7468092d8e01637f3efa8" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } } peers { permanent_uuid: "0c7d1b7607234fea8bbc5c770d3eee4e" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } } peers { permanent_uuid: "03ee39c597fe4dc5bec573b11c3eabf5" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } }
I20260808 04:39:58.196334 15774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c7d1b7607234fea8bbc5c770d3eee4e, State: Initialized, Role: FOLLOWER
Aug 08 04:39:58 dist-test-slave-drl0 krb5kdc[15558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.126@KRBTEST.COM
I20260808 04:39:58.196902 15774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3214d734db7468092d8e01637f3efa8" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } } peers { permanent_uuid: "0c7d1b7607234fea8bbc5c770d3eee4e" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } } peers { permanent_uuid: "03ee39c597fe4dc5bec573b11c3eabf5" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } }
I20260808 04:39:58.198568 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.187259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36666 (local address 127.15.228.125:38507)
0808 04:39:58.187392 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.187397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.187412 (+ 15us) server_negotiation.cc:407] Connection header received
0808 04:39:58.187542 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.187545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.187598 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.187686 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.188241 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.188865 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.189960 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.190309 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.190387 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:58.193371 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:58.193378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:58.193384 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:58.193418 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:58.195218 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.196417 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.196425 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.196428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.196701 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.197316 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.197320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.197322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.197658 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:58.197747 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:58.198194 (+ 447us) server_negotiation.cc:299] Negotiation successful
0808 04:39:58.198345 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 04:39:58.198904 15774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:58.199157 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.187172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.125:38507 (local address 127.0.0.1:36666)
0808 04:39:58.187346 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:58.187360 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:58.187440 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:58.187698 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:58.187700 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:58.187999 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:58.188118 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:58.188124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.189001 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.189007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:58.189816 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:58.189826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.190541 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.190544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:58.190646 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:58.191227 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:58.191241 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:58.193196 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:58.195361 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:58.195365 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:58.195369 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:58.196152 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:58.196903 (+ 751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:58.196910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:58.196915 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:58.197147 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:58.197840 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:58.197842 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:58.197965 (+ 123us) client_negotiation.cc:769] Sending connection context
0808 04:39:58.198732 (+ 767us) client_negotiation.cc:240] Negotiation successful
0808 04:39:58.198905 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260808 04:39:58.200021 15889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3214d734db7468092d8e01637f3efa8" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } } peers { permanent_uuid: "0c7d1b7607234fea8bbc5c770d3eee4e" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } } peers { permanent_uuid: "03ee39c597fe4dc5bec573b11c3eabf5" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } } }
I20260808 04:39:58.200322 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.188389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36725 (local address 127.0.0.1:46550)
0808 04:39:58.188576 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:58.188592 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:58.188671 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:58.189362 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:58.189365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:58.189547 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:58.189659 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:58.189665 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.190689 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.190693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:58.191670 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:58.191682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.192340 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.192344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:58.192515 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:58.192978 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:58.192992 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:58.194493 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:58.198009 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:58.198012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:58.198014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:58.198267 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:58.198596 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:58.198599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:58.198601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:58.198690 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:58.199777 (+ 1087us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:58.199781 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:58.199888 (+ 107us) client_negotiation.cc:769] Sending connection context
0808 04:39:58.199997 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 04:39:58.200152 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":3}
I20260808 04:39:58.200363 15876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } has no permanent_uuid. Determining permanent_uuid...
I20260808 04:39:58.200975 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.189051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.15.228.124:36725)
0808 04:39:58.189152 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.189155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.189167 (+ 12us) server_negotiation.cc:407] Connection header received
0808 04:39:58.189198 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.189200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.189237 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.189774 (+ 537us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.189859 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.190545 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.191836 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.192195 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.192746 (+ 551us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:58.194646 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:58.194651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:58.194654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:58.194677 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:58.197842 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.198400 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.198403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.198405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.198489 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.199433 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.199436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.199438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.199647 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:58.199764 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:58.200668 (+ 904us) server_negotiation.cc:299] Negotiation successful
0808 04:39:58.200796 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 04:39:58.202551 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.187492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55464 (local address 127.15.228.126:36939)
0808 04:39:58.187615 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.187621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.187819 (+ 198us) server_negotiation.cc:407] Connection header received
0808 04:39:58.188009 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.188012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.188057 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.188110 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.189520 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.190245 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.191900 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.192224 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.192329 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:58.197668 (+ 5339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:58.197672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:58.197674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:58.197705 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:58.199603 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.200828 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.200832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.200834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.200927 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.201452 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.201455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.201457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.201683 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:58.201908 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:58.202296 (+ 388us) server_negotiation.cc:299] Negotiation successful
0808 04:39:58.202418 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":49}
W20260808 04:39:58.204409 15903 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:39:58.204480 15903 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:39:58.205968 15666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8: Bootstrap starting.
Aug 08 04:39:58 dist-test-slave-drl0 krb5kdc[15558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163998, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.228.124@KRBTEST.COM for kudu/127.15.228.124@KRBTEST.COM
I20260808 04:39:58.207016 15679 master_runner.cc:186] Error getting master registration for 127.15.228.126:36939: OK, instance_id { permanent_uuid: "e3214d734db7468092d8e01637f3efa8" instance_seqno: 1786163997647470 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 04:39:58.210798 15679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:39:58.211378 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.200894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.228.124:36725 (local address 127.0.0.1:46564)
0808 04:39:58.201782 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0808 04:39:58.201800 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 04:39:58.201915 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 04:39:58.202279 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 04:39:58.202281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 04:39:58.202452 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0808 04:39:58.202542 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:58.202549 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.203711 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.203716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:58.204454 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 04:39:58.204465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.205214 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.205217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 04:39:58.205285 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:58.205790 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 04:39:58.205805 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 04:39:58.207456 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 04:39:58.209452 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:58.209456 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:58.209458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:58.209833 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:58.210155 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 04:39:58.210159 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 04:39:58.210161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 04:39:58.210260 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 04:39:58.210714 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 04:39:58.210718 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 04:39:58.210828 (+ 110us) client_negotiation.cc:769] Sending connection context
0808 04:39:58.210975 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 04:39:58.211136 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801}
I20260808 04:39:58.211403 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.201135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.15.228.124:36725)
0808 04:39:58.201257 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.201263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.201849 (+ 586us) server_negotiation.cc:407] Connection header received
0808 04:39:58.202050 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.202055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.202101 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.202146 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.202851 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.203588 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.204656 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.205010 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.205087 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:58.207604 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:58.207607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:58.207610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:58.207637 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:58.209325 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.209954 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.209957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.209959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.210043 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.210371 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.210375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.210377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.210597 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:58.210790 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:58.211040 (+ 250us) server_negotiation.cc:299] Negotiation successful
0808 04:39:58.211229 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":53}
I20260808 04:39:58.214818 15666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:58.215107 15876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5: Bootstrap starting.
I20260808 04:39:58.215935 15666 log.cc:824] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8: Log is configured to *not* fsync() on all Append() calls
I20260808 04:39:58.201635 15889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:39:58.218012 15876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5: Neither blocks nor log segments found. Creating new log.
I20260808 04:39:58.218501 15666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8: No bootstrap required, opened a new log
I20260808 04:39:58.218899 15876 log.cc:824] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5: Log is configured to *not* fsync() on all Append() calls
Aug 08 04:39:58 dist-test-slave-drl0 krb5kdc[15558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786163997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.228.125@KRBTEST.COM
I20260808 04:39:58.221437 15876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5: No bootstrap required, opened a new log
I20260808 04:39:58.223189 15666 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3214d734db7468092d8e01637f3efa8" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } } peers { permanent_uuid: "0c7d1b7607234fea8bbc5c770d3eee4e" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } } peers { permanent_uuid: "03ee39c597fe4dc5bec573b11c3eabf5" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } }
I20260808 04:39:58.223454 15666 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3214d734db7468092d8e01637f3efa8, State: Initialized, Role: FOLLOWER
I20260808 04:39:58.223923 15666 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3214d734db7468092d8e01637f3efa8" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } } peers { permanent_uuid: "0c7d1b7607234fea8bbc5c770d3eee4e" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } } peers { permanent_uuid: "03ee39c597fe4dc5bec573b11c3eabf5" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } }
I20260808 04:39:58.225383 15666 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:58.225703 15876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3214d734db7468092d8e01637f3efa8" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } } peers { permanent_uuid: "0c7d1b7607234fea8bbc5c770d3eee4e" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } } peers { permanent_uuid: "03ee39c597fe4dc5bec573b11c3eabf5" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } }
I20260808 04:39:58.225944 15876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ee39c597fe4dc5bec573b11c3eabf5, State: Initialized, Role: FOLLOWER
I20260808 04:39:58.226004 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.211879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36668 (local address 127.15.228.125:38507)
0808 04:39:58.211990 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.211994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.212071 (+ 77us) server_negotiation.cc:407] Connection header received
0808 04:39:58.212239 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.212245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.212297 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.212383 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.213224 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.214030 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.215242 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.215623 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.215968 (+ 345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:58.220814 (+ 4846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:58.220819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:58.220823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:58.220859 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:58.222770 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.223784 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.223790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.223792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.223926 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.224269 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.224273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.224275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.224568 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:58.224817 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:58.225635 (+ 818us) server_negotiation.cc:299] Negotiation successful
0808 04:39:58.225798 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 04:39:58.226264 15876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3214d734db7468092d8e01637f3efa8" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } } peers { permanent_uuid: "0c7d1b7607234fea8bbc5c770d3eee4e" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } } peers { permanent_uuid: "03ee39c597fe4dc5bec573b11c3eabf5" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } }
I20260808 04:39:58.228739 15876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 04:39:58.230012 15910 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3214d734db7468092d8e01637f3efa8" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } } peers { permanent_uuid: "0c7d1b7607234fea8bbc5c770d3eee4e" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } } peers { permanent_uuid: "03ee39c597fe4dc5bec573b11c3eabf5" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } } }
I20260808 04:39:58.230152 15910 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:39:58.230345 16273 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 04:39:58.232638 15628 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e3214d734db7468092d8e01637f3efa8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55464
I20260808 04:39:58.232883 15628 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 04:39:58.233009 15628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:58.233210 15911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3214d734db7468092d8e01637f3efa8" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } } peers { permanent_uuid: "0c7d1b7607234fea8bbc5c770d3eee4e" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } } peers { permanent_uuid: "03ee39c597fe4dc5bec573b11c3eabf5" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } } }
I20260808 04:39:58.233409 15911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 04:39:58.234071 15928 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:39:58.234153 15928 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:39:58.235069 15628 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3214d734db7468092d8e01637f3efa8" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } } peers { permanent_uuid: "0c7d1b7607234fea8bbc5c770d3eee4e" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } } peers { permanent_uuid: "03ee39c597fe4dc5bec573b11c3eabf5" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } }
W20260808 04:39:58.236035 15941 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 04:39:58.236214 15941 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 04:39:58.236184 15573 master_runner.cc:263] No leader master found from master e3214d734db7468092d8e01637f3efa8
I20260808 04:39:58.236498 15573 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:39:58.237170 15732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e3214d734db7468092d8e01637f3efa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0c7d1b7607234fea8bbc5c770d3eee4e"
I20260808 04:39:58.237380 15732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:58.238488 15732 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3214d734db7468092d8e01637f3efa8 in term 1.
I20260808 04:39:58.238909 15597 leader_election.cc:302] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c7d1b7607234fea8bbc5c770d3eee4e, e3214d734db7468092d8e01637f3efa8; no voters:
I20260808 04:39:58.239245 15910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 04:39:58.239523 15910 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [term 1 LEADER]: Becoming Leader. State: Replica: e3214d734db7468092d8e01637f3efa8, State: Running, Role: LEADER
I20260808 04:39:58.239916 15628 leader_election.cc:288] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [CANDIDATE]: Term 1 election: Requested vote from peers 0c7d1b7607234fea8bbc5c770d3eee4e (127.15.228.125:38507), 03ee39c597fe4dc5bec573b11c3eabf5 (127.15.228.124:36725)
I20260808 04:39:58.239884 15910 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3214d734db7468092d8e01637f3efa8" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } } peers { permanent_uuid: "0c7d1b7607234fea8bbc5c770d3eee4e" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } } peers { permanent_uuid: "03ee39c597fe4dc5bec573b11c3eabf5" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } }
I20260808 04:39:58.240051 15848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e3214d734db7468092d8e01637f3efa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "03ee39c597fe4dc5bec573b11c3eabf5"
I20260808 04:39:58.240300 15848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 04:39:58.241813 15848 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3214d734db7468092d8e01637f3efa8 in term 1.
I20260808 04:39:58.246551 15946 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3214d734db7468092d8e01637f3efa8. Latest consensus state: current_term: 1 leader_uuid: "e3214d734db7468092d8e01637f3efa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3214d734db7468092d8e01637f3efa8" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } } peers { permanent_uuid: "0c7d1b7607234fea8bbc5c770d3eee4e" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } } peers { permanent_uuid: "03ee39c597fe4dc5bec573b11c3eabf5" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } } }
I20260808 04:39:58.246687 15946 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:58.247570 15784 master_runner.cc:263] No leader master found from master 03ee39c597fe4dc5bec573b11c3eabf5
I20260808 04:39:58.247639 15784 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 04:39:58.252286 15958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 04:39:58.253333 15958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 04:39:58.267416 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.245428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36682 (local address 127.15.228.125:38507)
0808 04:39:58.245556 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.245560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.255738 (+ 10178us) server_negotiation.cc:407] Connection header received
0808 04:39:58.255929 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.255932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.255985 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.256035 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.256917 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.257736 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.259014 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.259359 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.259434 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:58.262127 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:58.262131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:58.262134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:58.262166 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:58.264205 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.264999 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.265004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.265005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.265218 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.266023 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.266028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.266030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.266342 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:58.266471 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:58.267069 (+ 598us) server_negotiation.cc:299] Negotiation successful
0808 04:39:58.267233 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:39:58.271430 15848 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5 [term 1 FOLLOWER]: Refusing update from remote peer e3214d734db7468092d8e01637f3efa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:39:58.272241 15946 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ee39c597fe4dc5bec573b11c3eabf5" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:39:58.277055 15911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3214d734db7468092d8e01637f3efa8. Latest consensus state: current_term: 1 leader_uuid: "e3214d734db7468092d8e01637f3efa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3214d734db7468092d8e01637f3efa8" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } } peers { permanent_uuid: "0c7d1b7607234fea8bbc5c770d3eee4e" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } } peers { permanent_uuid: "03ee39c597fe4dc5bec573b11c3eabf5" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } } }
I20260808 04:39:58.277199 15911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:39:58.278383 15732 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e [term 1 FOLLOWER]: Refusing update from remote peer e3214d734db7468092d8e01637f3efa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 04:39:58.279086 15946 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c7d1b7607234fea8bbc5c770d3eee4e" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 04:39:58.279464 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.246006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46576 (local address 127.15.228.124:36725)
0808 04:39:58.246119 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.246123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.251670 (+ 5547us) server_negotiation.cc:407] Connection header received
0808 04:39:58.251711 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.251714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.251762 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.251834 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.264439 (+ 12605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.265243 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.269054 (+ 3811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.269373 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.269460 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:58.273375 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:58.273379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:58.273382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:58.273411 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:58.275365 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.277414 (+ 2049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.277417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.277420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.277517 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.277855 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.277859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.277861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.278118 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:58.278257 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:58.279171 (+ 914us) server_negotiation.cc:299] Negotiation successful
0808 04:39:58.279298 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 04:39:58.282262 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:58.245601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.15.228.126:36939)
0808 04:39:58.245716 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.245720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.251280 (+ 5560us) server_negotiation.cc:407] Connection header received
0808 04:39:58.251416 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.251419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.251477 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.251568 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.266998 (+ 15430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.267682 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.270148 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:58.270546 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:58.270621 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:58.273769 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 04:39:58.273773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 04:39:58.273776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 04:39:58.273803 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 04:39:58.276168 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.278607 (+ 2439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.278610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.278612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.278703 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 04:39:58.281177 (+ 2474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 04:39:58.281181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 04:39:58.281183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 04:39:58.281413 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 04:39:58.281547 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:58.281979 (+ 432us) server_negotiation.cc:299] Negotiation successful
0808 04:39:58.282121 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 04:39:58.283170 15889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3214d734db7468092d8e01637f3efa8. Latest consensus state: current_term: 1 leader_uuid: "e3214d734db7468092d8e01637f3efa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3214d734db7468092d8e01637f3efa8" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } } peers { permanent_uuid: "0c7d1b7607234fea8bbc5c770d3eee4e" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } } peers { permanent_uuid: "03ee39c597fe4dc5bec573b11c3eabf5" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } } }
I20260808 04:39:58.283298 15889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:39:58.290609 15910 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3214d734db7468092d8e01637f3efa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3214d734db7468092d8e01637f3efa8" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } } peers { permanent_uuid: "0c7d1b7607234fea8bbc5c770d3eee4e" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } } peers { permanent_uuid: "03ee39c597fe4dc5bec573b11c3eabf5" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } } }
I20260808 04:39:58.290773 15910 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:58.291314 15889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3214d734db7468092d8e01637f3efa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3214d734db7468092d8e01637f3efa8" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } } peers { permanent_uuid: "0c7d1b7607234fea8bbc5c770d3eee4e" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } } peers { permanent_uuid: "03ee39c597fe4dc5bec573b11c3eabf5" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } } }
I20260808 04:39:58.291457 15889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:39:58.291692 15959 mvcc.cc:205] Tried to move back new op lower bound from 7316127736909639680 to 7316127736794214400. Current Snapshot: MvccSnapshot[applied={T|T < 7316127736909639680}]
I20260808 04:39:58.294086 15963 mvcc.cc:205] Tried to move back new op lower bound from 7316127736909639680 to 7316127736794214400. Current Snapshot: MvccSnapshot[applied={T|T < 7316127736909639680}]
I20260808 04:39:58.297310 15946 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3214d734db7468092d8e01637f3efa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3214d734db7468092d8e01637f3efa8" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } } peers { permanent_uuid: "0c7d1b7607234fea8bbc5c770d3eee4e" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } } peers { permanent_uuid: "03ee39c597fe4dc5bec573b11c3eabf5" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } } }
I20260808 04:39:58.297443 15946 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8 [sys.catalog]: This master's current role is: LEADER
I20260808 04:39:58.298597 15911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3214d734db7468092d8e01637f3efa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3214d734db7468092d8e01637f3efa8" member_type: VOTER last_known_addr { host: "127.15.228.126" port: 36939 } } peers { permanent_uuid: "0c7d1b7607234fea8bbc5c770d3eee4e" member_type: VOTER last_known_addr { host: "127.15.228.125" port: 38507 } } peers { permanent_uuid: "03ee39c597fe4dc5bec573b11c3eabf5" member_type: VOTER last_known_addr { host: "127.15.228.124" port: 36725 } } }
I20260808 04:39:58.298740 15911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 04:39:58.299052 15961 mvcc.cc:205] Tried to move back new op lower bound from 7316127736909639680 to 7316127736794214400. Current Snapshot: MvccSnapshot[applied={T|T < 7316127736909639680}]
I20260808 04:39:58.308960 15958 catalog_manager.cc:1347] Generated new cluster ID: f39b71108abf4a6d9ca69c9362fd36ba
I20260808 04:39:58.309036 15958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 04:39:58.347036 15958 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 04:39:58.351162 15958 catalog_manager.cc:1504] Loading token signing keys...
I20260808 04:39:58.369863 15958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3214d734db7468092d8e01637f3efa8: Generated new TSK 0
I20260808 04:39:58.371090 15958 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 04:39:58.407536 15783 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:58.404080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36686 (local address 127.15.228.125:38507)
0808 04:39:58.404219 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.404224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.406198 (+ 1974us) server_negotiation.cc:407] Connection header received
0808 04:39:58.406420 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.406423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.406484 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.406565 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.407394 (+ 829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36686: BlockingRecv error: recv got EOF from 127.0.0.1:36686 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 04:39:58.409202 15673 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:58.404930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55474 (local address 127.15.228.126:36939)
0808 04:39:58.405052 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.405057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.408278 (+ 3221us) server_negotiation.cc:407] Connection header received
0808 04:39:58.408462 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.408465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.408517 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.408568 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.409088 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55474: BlockingRecv error: recv got EOF from 127.0.0.1:55474 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260808 04:39:58.410737 15884 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:58.405614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.15.228.124:36725)
0808 04:39:58.405745 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.405749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.409834 (+ 4085us) server_negotiation.cc:407] Connection header received
0808 04:39:58.410008 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.410011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.410063 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.410116 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.410600 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46588: BlockingRecv error: recv got EOF from 127.0.0.1:46588 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260808 04:39:58.415529 15783 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:58.414092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.15.228.125:38507)
0808 04:39:58.414217 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.414235 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.414482 (+ 247us) server_negotiation.cc:407] Connection header received
0808 04:39:58.414701 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.414704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.414798 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.414853 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.415414 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36690: BlockingRecv error: recv got EOF from 127.0.0.1:36690 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260808 04:39:58.421205 15783 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:58.419596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36702 (local address 127.15.228.125:38507)
0808 04:39:58.419703 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.419707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.420248 (+ 541us) server_negotiation.cc:407] Connection header received
0808 04:39:58.420454 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.420458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.420509 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.420562 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.421100 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36702: BlockingRecv error: recv got EOF from 127.0.0.1:36702 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260808 04:39:58.427520 15783 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:58.425704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36708 (local address 127.15.228.125:38507)
0808 04:39:58.425808 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.425812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.426460 (+ 648us) server_negotiation.cc:407] Connection header received
0808 04:39:58.426700 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.426705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.426755 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.426809 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.427401 (+ 592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36708: BlockingRecv error: recv got EOF from 127.0.0.1:36708 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260808 04:39:58.440111 15783 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:58.435118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36718 (local address 127.15.228.125:38507)
0808 04:39:58.435271 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.435275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.438737 (+ 3462us) server_negotiation.cc:407] Connection header received
0808 04:39:58.439092 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.439096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.439162 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.439223 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.439987 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36718: BlockingRecv error: recv got EOF from 127.0.0.1:36718 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260808 04:39:58.454604 15783 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:58.451973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36730 (local address 127.15.228.125:38507)
0808 04:39:58.452255 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.452262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.453113 (+ 851us) server_negotiation.cc:407] Connection header received
0808 04:39:58.453381 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.453385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.453456 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.453516 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.454459 (+ 943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36730: BlockingRecv error: recv got EOF from 127.0.0.1:36730 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260808 04:39:58.475036 15783 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:58.473810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36746 (local address 127.15.228.125:38507)
0808 04:39:58.473953 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.473958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.474129 (+ 171us) server_negotiation.cc:407] Connection header received
0808 04:39:58.474236 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.474239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.474303 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.474413 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.474874 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36746: BlockingRecv error: recv got EOF from 127.0.0.1:36746 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260808 04:39:58.511663 15783 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:58.509798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36758 (local address 127.15.228.125:38507)
0808 04:39:58.510132 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.510146 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.510339 (+ 193us) server_negotiation.cc:407] Connection header received
0808 04:39:58.510653 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.510658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.510743 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.510907 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.511537 (+ 630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36758: BlockingRecv error: recv got EOF from 127.0.0.1:36758 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
W20260808 04:39:58.580000 15783 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:58.578708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36774 (local address 127.15.228.125:38507)
0808 04:39:58.578891 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.578897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.579054 (+ 157us) server_negotiation.cc:407] Connection header received
0808 04:39:58.579235 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.579238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.579299 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.579396 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.579876 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36774: BlockingRecv error: recv got EOF from 127.0.0.1:36774 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260808 04:39:58.712172 15783 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:58.710737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36778 (local address 127.15.228.125:38507)
0808 04:39:58.710957 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.710963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.711086 (+ 123us) server_negotiation.cc:407] Connection header received
0808 04:39:58.711280 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.711287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.711341 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.711435 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.712028 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36778: BlockingRecv error: recv got EOF from 127.0.0.1:36778 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260808 04:39:58.972000 15783 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 04:39:58.970788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36780 (local address 127.15.228.125:38507)
0808 04:39:58.971024 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:58.971029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:58.971043 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:39:58.971100 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:58.971103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:58.971150 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:58.971248 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 04:39:58.971863 (+ 615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36780: BlockingRecv error: recv got EOF from 127.0.0.1:36780 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260808 04:39:59.206033 15903 catalog_manager.cc:1259] Loaded cluster ID: f39b71108abf4a6d9ca69c9362fd36ba
I20260808 04:39:59.206135 15903 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e: loading cluster ID for follower catalog manager: success
I20260808 04:39:59.208016 15903 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e: acquiring CA information for follower catalog manager: success
I20260808 04:39:59.209450 15903 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0c7d1b7607234fea8bbc5c770d3eee4e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:39:59.237703 15941 catalog_manager.cc:1259] Loaded cluster ID: f39b71108abf4a6d9ca69c9362fd36ba
I20260808 04:39:59.237844 15941 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5: loading cluster ID for follower catalog manager: success
I20260808 04:39:59.239770 15941 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5: acquiring CA information for follower catalog manager: success
I20260808 04:39:59.242256 15941 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 03ee39c597fe4dc5bec573b11c3eabf5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 04:39:59.491256 16054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:59.486512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.15.228.125:38507)
0808 04:39:59.486741 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:59.486745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:59.486761 (+ 16us) server_negotiation.cc:407] Connection header received
0808 04:39:59.486851 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:59.486854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:59.487055 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:59.487156 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:39:59.487536 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:59.488182 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:59.489210 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:59.489551 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:59.489721 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:59.490147 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:39:59.490607 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:39:59.490723 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:59.490910 (+ 187us) server_negotiation.cc:299] Negotiation successful
0808 04:39:59.491043 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":66,"threads_started":1}
I20260808 04:39:59.500052 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:59.495241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.15.228.124:36725)
0808 04:39:59.495512 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:59.495516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:59.495530 (+ 14us) server_negotiation.cc:407] Connection header received
0808 04:39:59.495617 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:59.495620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:59.495742 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:59.495856 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:39:59.496090 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:59.496653 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:59.497808 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:59.498036 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:59.498131 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:59.498705 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:39:59.499434 (+ 729us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:39:59.499616 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:59.499855 (+ 239us) server_negotiation.cc:299] Negotiation successful
0808 04:39:59.499926 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":100,"threads_started":1}
I20260808 04:39:59.507892 16054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:59.503536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36800 (local address 127.15.228.125:38507)
0808 04:39:59.503672 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:59.503678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:59.503988 (+ 310us) server_negotiation.cc:407] Connection header received
0808 04:39:59.504207 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:59.504211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:59.504284 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:59.504340 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:39:59.504715 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:59.505488 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:59.506599 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:59.506897 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:59.507059 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:59.507349 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:39:59.507436 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:39:59.507516 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:59.507675 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 04:39:59.507733 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 04:39:59.507948 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:59.504173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46610 (local address 127.15.228.124:36725)
0808 04:39:59.504294 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:59.504310 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:59.504558 (+ 248us) server_negotiation.cc:407] Connection header received
0808 04:39:59.504676 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:59.504682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:59.504745 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:59.504832 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:39:59.505107 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:59.505651 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:59.506701 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:59.507079 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:59.507183 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:59.507428 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:39:59.507524 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:39:59.507601 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:59.507749 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 04:39:59.507805 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 04:39:59.508966 16073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 04:39:59.503793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55484 (local address 127.15.228.126:36939)
0808 04:39:59.504058 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 04:39:59.504063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 04:39:59.504300 (+ 237us) server_negotiation.cc:407] Connection header received
0808 04:39:59.504409 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 04:39:59.504412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 04:39:59.504586 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 04:39:59.504703 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 04:39:59.505167 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:59.505994 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:59.507122 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 04:39:59.507475 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 04:39:59.507567 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 04:39:59.507991 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 04:39:59.508381 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 04:39:59.508481 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 04:39:59.508763 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 04:39:59.508826 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":106,"threads_started":1}
I20260808 04:39:59.513494 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 15573
I20260808 04:39:59.533177 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 15679
I20260808 04:39:59.544783 16273 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFaBKn/build/debug/bin/kudu with pid 15784
2026-08-08T04:39:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3559 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8027 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (330590 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST